 Welcome and thank you for joining me today for the release of my report, How You Say It Matters, Communicating Predictive Analytics Findings to Students. My name is Alejandra Acosta and I'm a Policy Analyst at New America. I'm so excited to have so many of you here today to learn about a topic that's very important, but oftentimes goes under the radar. Higher education has engaged in more data-driven decision making in the last several years and more and more institutions are using predictive analytics to help more of their students graduate and to help close equity gaps as well. And so in these efforts to retain more students, institutions are putting a lot of effort into gathering their data and cleaning it up, setting up the infrastructure to implement predictive analytics and putting in a lot of social and political capital in this process as well. But it can be tempting to just press play and let the magic of the predictive model happen and wish for the best. It's really after you press play that the action really happens. And so this is where end-users like academic advisors or faculty advisors, other staff who engage with students and use predictive systems. This is where they have to interpret the data and engage with the students and offer them the support that they need to stay on track for graduation. So as the use of data and predictive analytics continues to rise, guidelines and training on how to put this into action are necessary. We need to be able to use this tool and communicate the findings both effectively and ethically. And so this report is an effort to do just that. This will help both institutions and students reach their goal of graduation. And this was not planned, but especially since now higher education will be online in the foreseeable future as a result of the pandemic. How we communicate with students in this particularly difficult and sensitive time continues to be important and is even more important than before if I may argue, if I may say so myself. So with that for today's presentation, what I'm going to do is first I'll take some time to share some of the research from the report. I'll go over some high level behind-the-scenes work that should happen before you actually send messages to students about predictive analytics and the work that you should do with your team before you press send. I'll also talk about what a successful message should look like and what some of the pit falls of a message can be as well. And after I present my research, then we will have a great panel with folks from the California State University at Northridge, Georgia State University and Morgan State University. And our panelists will share their direct experiences with implementing and communicating around predictive analytics. So I'm really excited for you to hear from our panelists and excited to share the research with you today. So let's get started and next slide please. So I don't think I need to make the case of why communicating matters to all of you. You are all here and so you seem to have very clear that this is important. But I do want to just emphasize it for a moment and share that there is a lot of research in social psychology that shares that how we communicate has real consequences. And I think that we can think back to our own experiences when we receive feedback at work, for example, how it's shared with us can really affect how we receive it. And so this applies to how we communicate with students as well. We're sharing potentially sensitive information with our students and we want to minimize the harm that we do to them while also supporting them as much as possible. And so three primary reasons why communicating really matters, especially when it comes to predictive analytics. So communicating matters because it matters for student success, of course, we want to make sure that we're as effective as possible so that we can help as many as many students reach graduation as possible. And so we want to be effective and also ethical to help our students feel like they belong to our institution and that they can do what they need to reach graduation. Communicating also matters for an institution's investment in predictive analytics. Like I said, there's a lot of time and effort, social political capital that is spent on implementing predictive analytics. And so you want to make this worth it for the institution. And you want to make sure that you put in the time and effort that you need into communicating so that this investment is not ill spent. The investment can be derailed by poor messaging, and that also is another important reason why you need to put in the time for preparing for messaging around predictive analytics. And finally, communicating matters for educational equity, of course. A lot of our student populations like students of color, first generation students or students who are of low incomes often struggle with navigating higher ed or have fewer resources or don't even know what resources they can use to help them in their educational journey. And so the way that we communicate predictive findings can help our students reach their goal of graduation much more easily by showing them how to navigate the ropes and showing them what resources are available. We can also affirm a sense of belonging for these students, which is especially important. And so these are some of the high level reasons why communicating matters. Like I said, I think you all know that this is important, but just want to emphasize some of the reasons why this is important. And next slide, please. So now I'd like to give a high level overview of the science behind communicating. I encourage you to read the report to learn much more in depth about some of the concepts that I recommend for creating both an effective and an ethical message. But I won't go into detail about them today. So the goal that you are trying to achieve with communicating about a predictive system finding is you're trying to both increase a student's chance of success while minimizing that harm that could be done to them. And so by using both behavioral economics and social psychology, this intersection helps you reach that sweet spot in the middle of both an effective and an ethical message. Behavioral economics informs how to create an effective structure of a message. So concepts like when you should send a message and what kind of content you put in it, whether it's interactive or not, all of those things help you create a message that will help a student do something. And when you use social psychology, you're looking into ideas and research that is helping you minimize the harm. So here, social psychology helps us understand how the content of a message can affect a student's perception of themselves as a member of society. And so this is what helps us help a student feel like they belong, feel like they have the capacity to improve and stay in school. So like I said, I encourage you to look into these concepts more in detail in the report, which should be linked in the chat a little bit later. And next slide, please. So it can be really tempting to start contacting students once you have your predictive system in place and just start sending messages to everyone. But just like there's a lot of work that has to be done in implementing predictive analytics, there's work that needs to be done in order to set up your communication around it as well. So here I have a list of some things that institutions should do prior to pressing send. So first, it's creating a diverse team, a team that's going to create your messages if you don't already have prepared messages by a vendor. And a team that's going to help you make sure that your messages are both ethical and effective. And by diverse, not only do I mean social social characteristics of diversity like race or ethnicity, but I also mean professional diversity. So having a student, someone from institutional research and advisor, all of those folks in your team will make you much more effective. Another important point to look into before pressing send is making sure that you have the right messenger. And so who we receive a message from has a lot to do with how likely we are to take action from that message. And you should pick a sender who has a strong relationship to the student. So this can be anything from a school mascot to the student's advisor to a trusted resource center on campus. And taking the time to select this messenger is really important. You should also look into what the right platform or modality is for your students. So you can send messages through text message through email, campus portal, chalk on the sidewalk in the quad for all I care, although that's not going to have been anytime soon. But the important point is that as long as your students are responsive to the platform, that's what's going to make it most effective for you. The question is often is texting the best way to do this. And the answer again is just what are your students going to respond to. And this may require a switch in platforms at the institution, which is again a difficult change. But if the goal is ultimately student success, then it is probably worth it. You should also test your message beforehand. And this can be as casual as asking the student intern in the office if they think the message is okay to something more rigorous like a B testing. The setting the timing of messages is also important. So we want to send these at a time that is actually useful to students. And then training our end users. So this ranges from bias training or diversity, equity and inclusion training to how to interpret predictive analytics as well. And so doing all of this work behind the scenes before you actually press send will make sure that when you send messages to students about predictive findings, they are as high quality as possible. And we'll go to the next slide, please. Great. So here what I'm going to do is I'm going to show you what some of the key components are to a successful message. And then I'll cover some of the reasons why messages might fail. So here we see a message that I received from my university, New America University. And it reads, hi alle finals are coming. But it may not be too late to improve your grades tutoring can help. 60% of your peers found sessions at the tutoring center helpful sign up here. And there's a link. So the first thing that's good about this message is that it's personalized. Sure, it says, Hello, it has my name in it. But what I mean by personalized as well is that all of the information pertains to me directly. This isn't a mass message that was sent out to all students. I'm much more likely to respond and take action. If the information was meant for me. And so make sure that your messages about predictive analytics are specific to the student. In the end, that's what predictive analytics was for, right? Next slide, please. Your message should also be timely. So like I said, when we send our messages is important as well. Here we see that the message says finals are coming. And so this is reminding me that this really important deadline is coming. But it's also showing that this message was sent at a time that is useful to me. So the tutoring center may not be as helpful to me in the second week of school. Maybe it is. But since I'm having an issue, sending this information just in time is going to make it most effective. Next slide, please. And I think here, this point of making a message realistic and positive is particularly, particularly important. And I really want to highlight it. So when we're sharing information about predictive analytics, the information can often be sensitive. You're, you know, not doing so well in a class. You're you've missed an important deadline for X reason. And so that is kind of sensitive information. And you want to make sure that you're communicating it in a positive way. So students don't freak out. But you also want to balance that with being realistic. And so here in this message, we see that it may not be too late for me to improve my grades. This strikes the balance between both realistic and positive, because it tells me that, you know, there is a chance of me improving my situation. But it's realistic in that it says it may not be too late, not it's not too late. And so this tells me that, you know, there's some things that may be outside of my control that could affect whether or not I'm actually able to change my grades. And this could be each faculty member's policy or some other deadline that is not explained in this message here. And next slide, please. A successful message is also interactive. So as I shared, we're trying to get students to do something, to take action so that they can reach graduation. So here we see that at the bottom, the message includes a link. And this makes it extra easy for me to just click on the link and do the thing that you're asking me to do. There's hopefully no avoiding it, and I can just do it right away. You can also include instructions in your message. So that students know exactly how they should take action. This reduces the mental load that it takes to take action and just makes it much more likely for things to get done. And next slide. And last but not least, the message should be brief. So it's really tempting to want to share all of the resources on campus with the student, but nobody's got time to read your five text messages or your super long email detailing all of this information. So be direct and be concise. Next slide. Great. So those are all of the things that make up a successful message. And of course, there's a lot more in the report as well. And so you may be thinking, well, I've done all of these things. My messages still aren't working. What's going on? And so here I offer three reasons why messages may not actually work. And one of them is that perhaps you didn't conduct sufficient population analysis. Maybe you don't know your population well enough. And I'll offer results from a non higher ed related study where researchers were trying to get teenagers to eat more healthy foods. Some teenagers were told if you eat healthy, you'll live a longer life, you'll be healthier. But then another group was told that junk food companies spend a lot of money on trying to get you to buy this stuff. And so they tapped into the teenagers sense of rebellion and desire for rebellion to actually get them to eat more healthy foods. And that was the more effective group. So make sure you really understand your students so that you can communicate to them in a way that's relevant. And next slide. Another reason why messages may not work is you have little rapport with your students or the messenger may have little rapport with your students. So if you're receiving a message from an entity that you don't know or you don't trust, it's much less likely that a student will take action. So make sure that you do select somebody that the student has a good relationship with. And next slide. Thank you. And the last reason why your messages may not be working probably has nothing to do with the messages themselves at all. I believe that if we are not also taking a deep look at our institutions, our processes and systems, no matter how clean our data is, how extensive our predictive analytics data is, how accurate our algorithms are, or how fancy our messages are, none of that is really going to work if we have systems in place that are not helping our students. So if, for example, you're asking a student to go see their academic advisor, the student clicks the link and tries to do that right away, but there's no meeting available for like three weeks and that's not helpful for the student, then your message is not going to be as effective. And this obviously takes time and resources, but I think it's one of the real benefits of implementing predictive analytics is we not only help the student in the short term, but we hopefully look at our systems as well in the long term and make the changes that we need to make higher and more easy to navigate for our students. And next slide, that is all that I have to share from the research report. I do just want to highlight that again, as we're probably going to be online in higher ed for a lot longer, or at least for the foreseeable future, the way that we speak to our students in this particular particularly challenging time is going to be more and more important. And so I encourage you to share this information with your colleagues and not just the ones that have predictive analytics or are thinking about it, but really with everyone right now since we're all going to be online. And with that, I do want to switch over to our panel. I'm really excited to have a fantastic group of experts here with us to share their experiences in implementing predictive analytics and successful messaging. And I'm going to have my colleague Iris Palmer who has done a lot of our work on predictive analytics at New America as well have moderate the panel for us. So thank you, Iris. Well, thank you, Allie. And thank you so much for that parting word of wisdom about how relevant this is in the current higher education environment. I think that's really, really important to keep in mind. As Allie said, my name is Iris Palmer. I'm a senior advisor for higher education in the workforce here at New America and have done a lot of work on our ethical use of predictive analytics, of which this is a really, really important part. And Allie has done amazing work on it. With that, I'd like to introduce our panelists today. We have with us Dr. Infume, who's the assistant vice president for student success and retention at Morgan State University. We also have Dr. Adams, associate vice president for undergraduate studies at California State University, Northridge. And finally, not last but not least, Lindsey Fivefield, a chatbot project director at Georgia State University. And Allie will also be on this panel with us to sort of reflect on how the research she's done can inform some of the conversations and examples that we're going to hear about from our panel today. So I'd like to get started by asking a pretty broad question. What does predictive analytics messaging to students look like on your campus? And I'd like to start with you, Dr. Infume. Morgan State is an HBCU, so we'd like to hear a little bit about how you approach this with your student population in mind. Well, thank you so much. Because we are an historically Black college and university, we are very aware of risk, which is one of the keywords when you bring on any predictive analytics platform. We have two partners actually with EAD and Starfish. And it was very important for us when we already know that we are still 80 percent African American students and or the 60 percent of our students, our first generation students, always more than 50 percent, more so 60 percent of our students are Pell eligible students. And two thirds of our students need some type of development or remediation coming in the door. We already know and anticipate that our students are coming from the traditional risk categories, minority, low income, first generation students. So it's not, we came into it with the idea that we needed insights beyond the obvious risks. And we don't even like that word. The first thing we did was eliminate the word risk from our model, which then takes it out of the context of any communication. And so the platforms typically want you to see which students are at high risk, moderate risk or low risk. And we flipped that because we're less interested in the risk of the students and more interested in what we're going to do about it. What are our actions going to be? So now we have instead of high risk students, we have high priority students. You are a high priority to us as a community, moderate priority and lower. We don't want to say no student is a low priority student, but lower priority. So first we approached it just by changing our framework from a risk model to a priority student model. And then from there, we made sure that none of our risk scores actually show in any platform. So students are never made aware that they are high priority, moderate or low priority students. And we even took that away from advisors and faculty. It's a very select group of people who get to really understand whether a student is high priority, moderate or lower priority. So that's the beginning of our framework. And then from there, it's just making sure that whatever we're doing is for the benefit of students in taking a particular action or changing of behavior. And as Alejandra has already said, it's from a positive model of trying to provide support. And when we get into some of the other questions, I'll give you some more specific examples of exactly what I'm talking about. That's incredibly helpful. And some of the examples about how you target that particular population that you serve. Dr. Adams, I know that you serve a lot of transfer students and a very, maybe not a totally different population, but a slightly different population than Morgan State. Can you tell us a little bit about how you craft your messages for that population? Sure. So CSUN is a large, mostly commuter institution in the Greater Los Angeles area. And we are a HSI, Hispanic Serving Institution. And we have actually not a dissimilar population in terms of Pell, First Gen, and the like to Morgan State. But one thing that makes the CSU a little different than a lot of systems in the country is that more than half of our students are upper division transfers. We don't take lower division transfers. So there are students who have made it through the basic GE stuff and the basic lower division pieces of their majors at the California Community Colleges and then they transfer to us. The number of students who set out to do that and actually do that is pretty startling. It often is in the low teens in terms of students who actually get that thought lift. So when students actually make it to CSUN, they are ready to be successful. And one of the things that we discovered when we started using some of the predictive analytics models is that they were based on freshman completion and they weren't relevant for our students. So for example, one of our predictive analytics, we could check whether or not the students were going to graduate in four years, six years or ever. And that wasn't particularly helpful when we were talking about transfer students who we really wanted to figure out whether they were going to graduate in two years or four years. So we really had to separate the two groups and we could work from a much, much more positive frame of mind. We already know that these students are more likely to graduate than our incoming freshmen. We already know that they've been successful in college. And so part of our communication with them is about where are the pain points for transfer students rather than the pain points in general. So we can look at things like are there particular courses that they struggle in or those courses places that we want to communicate with them. And then we also want to make sure that they're giving information about the differences between the way that CSUN academic support structures work and CSUN academic rules work, then it might have been at the community colleges. So for example, knowing about our late ad drop process, knowing about our probation qualification process. And again, and we'll talk more in detail, but making sure that those messages are distinct depending on the student group because it's they they're not dealing with the same issues. Really important point. Thank you for that. So Lindsay, can you tell us a little bit about how you all are crafting messages at Georgia State, particularly given your very also very diverse population? Right. To kind of echo the rest of the panel, our population looks very similar, right? So over 55% eligible, 75% non-white. So I think kind of the first step in crafting the message is understanding your student population. A bulk of the messaging concerning pre predictive analytics to students at Georgia State really does come through the advisor channel. So that is the advisor understanding their population and then and the messaging out to the students. But to kind of echo too, it's rare that we communicate directly about the predictive analytics point. And instead, what will you leverage kind of what you're seeing from the predictive analytics to speak to the student? Because if you start saying predictive analytics to the student, they're not hearing that, right? So I think it is much more of a successful tool for the staff of Georgia State or of any institution to use to prioritize which students you're reaching out to. Which students are you reaching out to first? Which students are you leaning into this resource and to really kind of particularly as that front-end advisor unit, how are you leveraging what you're seeing in the predictive space to lean the student to success and communicate that effectively to the student? Again, it's rarely directly communicated to the student, but much more used as a tool and then kind of translated across a desk in person to the student in terms of like, you know, why would this be concerning that I didn't do well in this course? But in general, not just, yes, we use the advisors to communicate predictive analytics to students, but at Georgia State, we've really kind of integrated the predictive analytics model and the data-driven model kind of across all of our success platforms. And so we really kind of use it as an integration tool in terms of how can we leverage all of the technologies and platforms and programs that we have to address the students who are in most need. You know, we've seen that even with our chatbot technology, which is separate from our predictive analytics, how do we take what we're seeing on the predictive analytics side and integrate that into another tool to then feed back into kind of that in game of getting a student sitting across from the person who can most help them. So, yeah, it is a much more of an integrative approach, but a lot of the messaging specifically about the predictive analytics happens on a one-to-one student advisor type interaction. Thanks for that, Lindsay. Alejandra, do you have anything you wanted to add? Yeah, I think this just, sorry, I have AC on. I think this just highlights the importance of really getting to know your student population. So all of these institutions are very diverse and students have needs that really go beyond the box that you receive or the package that you receive of predictive, like of a predictive analytics system. And so I think that highlights the importance of that and also the importance of the squishy social psychology stuff that I think we often like to cast aside because it doesn't have to do with the quantitative part of predictive analytics. But with all of these students, if we're not careful with how we communicate with them, we risk communicating our biases to them and putting them at risk for stereotype threat and self-fulfilling prophecy. So I think this also highlights the importance of both the effectiveness of behavioral economics and getting a student to do something, but also the squishier stuff that I could argue maybe is even more important to make sure we're not harming our students. Yeah, that human factor is so important and it actually leads us to our next piece. So I think Lindsay touched on a little bit how a lot of this communication actually happens very personally between an advisor and an advisee with some data informed pieces to it. But I do know that there are larger messages that go out to the institutions and I was wondering, Dr. Adams, if you could talk a little bit about how you craft those messages and sort of what you think about as you think about any other kinds of communications to students that maybe aren't that personal face-to-face with the advisor. Well, one of the things we did, and I know a lot of campuses have done this, is that we really went back and looked at our sort of standard communication, for example, what the letter that students who were put on probation looked like, what the disqualification letter looked like. That was our sort of our first step and we realized how negative and sort of deficit-minded the language, especially for the probationary letter was. So we rewrote it and I think it's a lot better but that then prompted us to look at all of the messaging that we do broadly so that we really tried to get away from you haven't done this right into, hey, how can we help you so that those were at the sort of macro level. We also have a chatbot and we found that to be an incredibly helpful way to communicate both in a targeted way and in a broad way because the chatbot has a sort of different communication style. It doesn't pretend to be anything but a chatbot and it's a little jokey and it's a little funny and it uses gifs or gifs, I don't know which people have different ways of pronouncing that, but anyway, and ours is called c-sunny and so it has a sort of cheeky way of communicating that I think puts students at ease and then when we actually have humans communicate with them, whether it's an advisor or a faculty member or the touring center, then you see the communication shift a little bit but we are still trying to use that positive framework instead of the deficit-minded framework and I really do think it's made a difference. I mean these are just anecdotal but the probationary letter has my name on it and every semester since we changed it I get a handful of students writing me and telling me why they weren't successful and thanking me for helping them sort of get pointed in the right direction and I've been doing this job for eight years and until we changed the letter, I never ever got responses from students in response to the probation letter so we're at least reaching some of them and that's actually one of the real big keys and that is that not every communication strategy is going to work with every student which means that you've got to use lots of different communication strategies so that you can get at pockets of them right so if you can get it five students with your new probationary letter and a hundred students with the chatbot then that's a hundred and five students you didn't have listening to you before. Dr. Adams just the follow-up question, do you customize the language the chatbot uses or is that sort of come with the package that we do both you know so the vendor I think does a terrific job and I know George State uses the same vendor we do the vendor does a terrific job with giving that sort of baseline way of communicating but then what I did when we instituted the chatbot was that I went out and I hired an employee whose expertise was in writing right so she's a rather than getting a technical person I got somebody who was a creative person so that she could make sure that when she wrote our customized messages that they echoed the the vendor delivered messages so it's definitely a mix and you need both and you need to make sure that the the bot is consistent in the way it's communicating and then the humans are don't have to be consistent the way they're communicating other than that they need to operate from an asset driven perspective. It's really helpful. Lindsay I know if we want to follow up on that you I believe George State does a lot more customization of their chatbot language can you talk a little bit about how you create and test that? Yeah sure just to speak kind of piggyback off of Dr. Adams we do customize the language inside of our bot we also customize the message and target the messages that are coming out of our bot right so we use that kind of of a leverage to make sure that we are we're sending a message that the students are going to listen to right. We have worked really hard to create a personality in our bot much like C Sunny our bot is actually named for our mascot Pounce and who is a a giant blue panther and we wanted the students to see the chatbot as as a peer but a much smarter peer right like the peer that you always can go to for you know they always know what's happening and they're always on top of everything so and I think that's kind of opened up students to be more receptive to communication coming from the department right so they know they kind of have this a friendly personable very relatable type of environment to talk to about institution type things that then they can kind of take that information and find the right office and get to the right person a little bit faster to be able to communicate whatever it is institutionally that's kind of trying to be sent to them so we've also leveraged our chatbot in conjunction with emails right so we know like an email is about a campaign's about to go out through advisement we will either we'll work with them what makes sense do we send this nudge prior to that do we send this nudge after that what kind of like insights can we work together to kind of collect you know collectively push the students to take the action that we want and the same breath to keep them encouraged and understanding like this this is to help you this is to move you forward we we've referenced emails in our in the chat right so they're seeing if their perception of counts is very peer related having your your friend say hey did you get that email from the professor you're a little bit more inclined to go check your email so that in the same vein we'll say hey your advisor should have sent you an email right check your email for updates on this or whatever it is so and that that not only does that help the student feel like that message is directly for them very personal but it also helps them feel like this isn't just the advisement office working for me this is Georgia State kind of trying to help me move forward so that's a really great example um Dr. Fumme I know that you you you mentioned starfish and I know you have customized some of the messages that come out of that can you talk a little bit about how you've done that but you're on mute thank you that never gets old does it so one of the things that we've done is to really track the open rates for our email so we're primarily still emailing not text messaging and we have our bot but we've worked really hard to build in the culture that email is our official mode of communication between students and the university and so trying to break through the noise of all the many emails that go out we track the rate at which for any message we send it's opened um if there's a link is the link clicked on is the message responded to um and then yes it's important to see if we were referring them to a certain office did they actually have an appointment but the main thing is you know our students even clicking opening paying attention to our messages and the number one thing that we've found as far as the click rate the open rate the response rate is the subject line it all comes down to the subject line of an email we figured out if you don't have the subject line right it doesn't matter how well as Alejandra said earlier that it's concise it's positive it's personalized you have to get them to the message to even find out that is concise and personalized and positive and all like that so um the success we've had in that click rate is about the subject lines typically that pose a question that are very intriguing um if we're dealing with something that has to do with payment um one of the predictors of our student success is is financial clearance do students clear are they going through the billing and financial aid process in a timely fashion so when we're sending out messages about the payment process we come directly with do you need more financial aid well hello yes click here you know do you need a payment plan sure do let me see what that's all about um and then we use more general ones when it's about an academic concern um do you need help can we get you a tutor um but questions do very well where the students really want to click to get the answer um also again things that can be somewhat misleading but very ethical as if like something really exciting is going to happen if you click here and it may not be as exciting as they thought it's more so of a of a service that we're trying to provide but the the subject line of our messages and our emails are really important the other thing is um we use that strategy and behavioral economics that Alejandra talked about uh which is comparing students to other students you know you know only 10 percent of you uh haven't selected your course you know 90 percent of students have already done this you know it's it's not too late or did you know that when you do this you're much more likely to earn a grade of a or b um so lots of that comparative um analysis with what other students are doing has helped us um in our behavioral economics kind of framework to our messages and then um she also used in her report one of my personal favorite examples about how students actually pay attention to the language um and if it's we were actually being too positive in some of our messaging where faculty felt like we were misleading students to believe that things weren't as bad as they looked um when we were saying hey you can still improve your grade it's not too late well we have a great feedback process to get other stakeholders faculty to tell us you know how they think our messaging is going from a student's success standpoint and several faculty said you're over promising you know students are coming to us with these messages saying hey i got this message that says i can still get a good grade i can still pass it's not too late and what faculty were saying it's it's not our place to promise whether or not it's too late and so we kind of dialed back the language just a bit um that and she used a specific example um just adding in the word may not instead of not made our faculty feel much better it may not be too late um you may still be able to submit assignments you know it's possible that you could improve your grade you know as opposed to you can still improve proof your grade so a lot of um the devil is in the details just making sure that you know while we're being positive we don't you know over promise and that we're not misleading students unless it's a catchy subject line and we ride the line to make it exciting alahandra did you have anything to add from the behavioral science perspective i think um dr kume gave you a really nice shout out there yeah so thanks so much for sharing that example with us a few months ago i thought it was um so insightful um but what i what i want to highlight here is is not so much the science of it but the the fact that the process of communicating and the use of predictive analytics is iterative so this isn't this isn't a one and done i cleaned up my data i crafted my messages press play press send and we're done it's a going back and always reflecting on how things are going and being um very intentional about looking at the data looking at the results of your actions and always being flexible and changing this to really fit what students need the most thank you for that um so i'm gonna go ahead and open it up to audience questions because we already have a couple and they're pretty interesting so um the first one is uh what is the trade-off between many different kinds of messages to address different preferences of students and overwhelming students with lots of messages um i i think this is a very common issue i'm wondering if anyone can tell us how they address this dr adams and then lindsay looks like yeah i mean boy this is absolutely a tough one and it requires a lot of coordination um uh one of the things that we found uh in trying to to to sort of layer communication is that we have to make sure we're not uh over communicating with them especially via text um the emails you can be a little more um uh often though you know the more often the less likely they are to read them um but if you text them too often they'll opt out and then you've lost them um and so it's really critical that you have somebody who's managing a holistic communication structure um so that you're not texting them more than once a week or so you're not emailing them more than a couple of times a week um and that it it varies throughout the semester so that you're you know you're not inundating them it's fine to do it a little more at the beginning um and a little more at the end but you know you also want to give them a little bit of a break uh in the middle so yeah it takes some coordination and of course you've got multiple offices over multiple divisions that have to uh coordinate this um but it you know having having that structure in place putting that structure in places is really critical it you know i'll go back to something dr mufume said a few minutes ago which is you also have to be really careful about who has access to all this stuff right so you have to be careful about who has access to the risk models you also have to be careful about who has access to the texts um and you know not everybody should be allowed to use the bot um and and so it it allows us to use it in a more strategic way let's see yeah um and that's very similar to kind of what we do at Georgia State so we very much let emails kind of come from the different departments be the official you know connection word from Georgia State but then we then we kind of layer on our chat bot to help prioritize the student's attention does that make sense so if if if we have a planned message to go out about a study abroad initiative but a student is is living in a place where they have a critical uh academic or financial hold then the students and we're not going to message about the the study abroad we're going to message and kind of keep the attention on on what is the most critical issue um most and most acute for the student to kind of progress forward so um the chat bots really helped us keep the attention of the students i think on the right thing right because they are going to get emails from financial aid they're going to get emails from advisement from the study abroad office from their program but i i think layering in the text the text nudges that really can direct their attention to what is most critical for their progression has really kind of helped our students pick up a little bit of speed when it comes to to moving towards graduation yeah doctor yeah i i think i want to go back to something alahandra said in her presentation and that's about the rapport that students have i feel that that's directly related to how frequently is a good you know what's a good rhythm of communication weekly daily it depends on who you are and we our students we have students on all of our committees that kind of deal with the different technologies that we use as you said we have our student workers when all else fails who who tell us oh dr fume you can email us we read your emails we just don't want emails from these other people and so the the issue for us is how to stay on the list of your you're the cool kids like messages from you good messages from this office we ignore and i think it's it's not it's it's it's the uh rhythm and the rapport that when students click on our emails that they that they feel that they're helpful they're not irrelevant oh like i'm so glad i got this information oh and they actually tell other students make sure you open that email about this or that says do you need a payment plan or can we help you or whatever and i think that as long as students are feeling that you're not wasting their time and the messages aren't too long and they're not too wordy and they're not negative then you can be um perhaps more you know purposefully redundant um and then there are other offices you know i let's let's be honest we do the same things you know we get our campus announcements and there's certain ones that catch my you know attention and i want to make sure i don't miss those versus other ones it's it's just how to be on the these list not the those lists and i so for me that's what determines the frequency and whether it's okay and if it's redundant you as long as they you know you haven't gotten to the point where they're they're rejecting um what you're offering then i think you're safe so i think that's such an interesting point we hear a lot about how the horse is out of the barn with email and everyone has the email list so it's too crowded and students don't see it and we have to prevent the same thing from happening with chatbots and text messages which i think is all true but i love the way that you dr mpumi have thought about how to make your own email stand out from the rest of the um noise i think for lack of a better term it's really interesting um so another question we received is um um do the sort of uh cheeky gifs gifts uh components of messages ever backfire they can be really hard for us old people to keep up on what's going on what's appropriate what's not how do you make sure that your tone is appropriate while also relating to the student population that's changing so quickly who wants to go first on this one i think george estate is the most creative in this space i think lindsay should answer first because i feel like their the pounce is so like creative and fun and cool we're not there yet we're pretty traditional so i think lindsay well thanks um yeah we really do try to stay relevant to the students but with a with a student body that is as large as ours we're you can never please everybody is a reality so so you're trying to cast a wide net um one of our more interactive um campaign through the chat bot was actually when we're we sent it out last december and uh it had baby yoda in it right and it was it was essentially like baby yoda with a with a santa hat and it was like hope you have a good holiday but it was actually asking about registration and so like the the question the the meme kind of caught their attention um but students really seemed to like that um another way that we've really kind of found success with is if you if you there is a balance there right and you don't always want to when you feel like you're starting to kind of get reiterated and what you're communicating that's when you leverage the meme that's when you leverage the gift right to to really kind of say hey you know i'm trying to for you to hear me right and if you're hearing memes right now then how about this meme right how about this Jeff um one of the success stories that we have with our chat bot is that through the admissions office they did a campaign on movie trivia right because who doesn't love what what 20 year old doesn't love movie trivia right and so they did a campaign on movie trivia that related to Georgia State but what was striking about that was the next message they sent was FAFSA and that FAFSA then that next FAFSA message had the highest click rate on the FAFSA that they had had yet so um so kind of not just thinking about it as this one image this one you know you know cheesy cheesy meme or cheesy gift um but really as a piece in a larger communication puzzle right and how all those things start to work together to how the students perceiving this communication coming from the office so um we try very hard you know to make financially fun and uh you know not kind of scary and a lot of a lot of what we do kind of with the chat bot and messaging these things is to take complicated issues and boil them down to very achievable steps for the student right um and if i can message you that in 160 characters then then you could do this you need to call this office and do this thing right and if that doesn't work then here here is a here's a gif that uh probably demonstrates how you feel about it so Lindsay are you just that cool or do you test them with students oh no we are me personally for sure not that cool but um we actually have a team of graduate assistants and a younger staff member that that works to keep us the coolest um but a lot of our other offices across campus implement the same strategy kind of bringing students to the table um to help inform the message what does that look like how can the is this cool today this not cool today um the yoda the yoda um meme we used was actually crafted by a student so um you know we they are a part of the conversation to make sure what we say um sounds cool because it's me personally not so much so this actually leads us to another question um and that is we talk a lot about customizing the communications to certain types of student doctor atoms you talked about how it's different for transfer students i'm talking to me you talked about how it's different for your particular student population um how do you avoid profiling students um when you're communicating what can be difficult information um in a more high level way or do you sort of leave that communication up to your advisors i think one of the more interesting things that that we've done is is to uh involve the faculty um one of the things that we had our institutional research office sorted was that uh our we have a problem with freshman retention no surprise um and we sorted out that um if a student got a d or an f in a class their first year they were less likely to be retained and then we went looked see which classes they were most likely get a d or an f in um and you know it turned out that you know 80 percent of the df df's and us were in 10 percent of the classes so anyway what we did was we went to the faculty in those classes and said hey can you help us identify students uh who are struggling early we did not use the predictive analytics we used the actual performance of the students in the classes and we discovered that if we let students know uh in the second week that they were struggling or the fifth or sixth week that they were struggling that they actually were pretty likely to turn it around once you got past the sixth week and this goes back to what Dr. Mifume was saying that you know you can't promise their grades but um once you got past the fifth of the sixth week it that it was harder for them to turn it around but so we did look to see where the intersections were between the students that were struggling in these classes and their concern level um but we focused on what was happening in the class you know Professor Adams says that I actually used it in a class I was teaching that you're not coming to class and then an advisor followed up with them and the faculty member followed up with them and so it ends up being a sort of whole culture of care and you're you're still using the predictive analytics but that's not where the message is coming from the message is coming from you're not doing well in this class and we want to help you turn it around now and the students are really surprised when you contact them in the second week um but that often uh shocks them into oh I ought to actually focus now uh in a way that that uh it might not if it was just sort well you're a high concern student so really I just wanted to jump in with the with an example from this semester to what Dr. Adams was just saying about getting that live class in person class feedback we do two progress surveys through starfish this semester one a couple weeks before midterm and one a couple weeks before final so that faculty have an opportunity to check in with students on their progress so that they you know can study and focus before those exams this past semester after we had all shifted our students didn't come back from spring break and we've gone virtual and remote for the balance of the semester I asked faculty um there's always a pre-communication that goes out to faculty um that we use kind of as our as our chance for student success to send a message twice a semester on what we're focusing on and and this time we said look it's a it's a pandemic students are nervous there's a lot of anxiety we made a bunch of policy changes past fail grades extending the date to withdraw let's use this progress survey in a way that we never had to give students a different kind of feedback that's more focused on how they're doing at home how are they adjusting to the virtual remote environment let's not just make this the run of the mill progress survey let's go ahead and give more kudos to to affirm students who are doing well in the at-home environment versus those who really may need to request a pass fail option or are struggling they really need to to know that and so what happened is after that progress survey and we got our usual like 9 000 data points that are from faculty not automated um the deadline for pass fail you had to apply for a pass fail grade they got hundreds of students who as a direct result of the feedback that they got through that progress survey made some really hard decisions about pass fail and withdrawing from classes so we customized it in the sense that we tried to make it about the current situation that students found them themselves in because of uh COVID-19 that is an amazing example uh thank you so much for that and we are out of time so I think that's a lovely um anecdote although you know I don't like our situation but it's a good anecdote to end on and thank you all so much for joining us this was a really amazing conversation I wish we had another half an hour but you know it is what it is and uh thank you all so much thank you everyone for joining us and have a great rest of your day