 Hello, everyone. Welcome to our session. We're going to start. I hope we have more registers, actually we have 65 registers, so I'm sure people will show up. My name is Green Bugida, Dean of Libraries at Stony Brook University. Our talk is ChatGPTAI in Higher Education. You know, when I proposed this talk, I was not expecting this explosion. Every day, every day, there is a new thing and it's really kind of shaking our world, whether higher education or personal life, teaching, learning, or your daily life. So this is really, really a popo. So I'm going to be the last. So brief introduction. We have really, really distinguished speaker with us. We have Susan D'Agostino. She's a science writer and scholar and mathematician, former faculty. If you follow inside her head, so she's the one. So she has really that broad perspective. And then next is Peter, Oganichak. I hope I didn't butcher your name. I met Peter like 11 or 12 years ago at JCDL. He's a DH data scientist, scholar in information science and also AI. You will hear really good stuff about, like, how we use GPT, even before GPT. And then we have Boris Zhang, NLP expert within the University of Florida Libraries. As far as I know, she's the only NLP librarian in the US. And tell me if I'm wrong. We need more people like her. And then I'll present at the end. My talk will be more over the geopolitics of chat GPT at the macro level. So welcome again. Please ask questions, comments, because I'm sure you have many. So Susan, and please say something about you. Thank you so much for coming. It's great to see you all. My name is Susan Dagestino. I'm the technology and innovation reporter at Inside Higher Ed. And I'm really glad to be here part of this conversation. I show up at conferences because I learned from all of you. And so stop me in the hallway, have coffee with me. I'd love to meet you. Okay, so what I'd like to do today is tell you a story because I'm a storyteller. How did I get to Inside Higher Ed and reporting on technology? And how did I arrive at this moment? And what am I doing right now? And I think the best way is probably, well, let me begin a little bit by telling you my background. So I've actually been steeped in higher ed for several decades. I'm trained as a mathematician, as Karim pointed out. And I was a professor for a decade, 10-year, taught my heart out, loved it. I was at an open access regional university that I took great pride in teaching in that setting. I always loved writing. And I thought if I'm at some point, I need to do writing more seriously. So I left teaching about five years ago and pivoted my career to journalism with the idea that my interest in math and science, I could contribute and teach in a different way that it would be public facing math and science literacy. In my job at Inside Higher Ed, I talk with students, professors, administrators, librarians, of course, staff members, and they actually inform my stories very much. And I'm grateful for all of my sources. Many of my stories have typically at least three people quoted in them from the higher education community, but often I might speak to 15 or 20 even before I actually get that story right. I meet people at conferences, and I also call them up and cold call or email typically first and then set up a time to talk. And next year I'm actually going to be headed to Columbia University. I will still be publishing Inside Higher Ed, which I'm very excited about, doing some more long form pieces specifically on the topic of artificial intelligence and the disruption and opportunity that that presents in higher education. So what I've been writing lately, if you've followed Inside Higher Ed or any of my stories, I have short, medium, and long term stories, but many of them are short term and I turn around stories pretty quickly. But next year I'll be focusing specifically on the topic of this panel and narrating the conversation as we go, that first draft of history as we call journalism. So let me, again, share a little bit of a story here with you. So if we want to back up until last fall, so this was my first story in Inside Higher Ed, Computer Sciences Challenges as seen by its pioneers. Every year I attend the Heidelberg Laureate Forum. I have for I think the past four or five years, and I will this year again. This is an invitation-only meeting with all of the Turing Award winners. They are the equivalent of the Nobel recipients in Computer Science. They are the ones who are responsible for all of the technology that powers our lives. And they are deeply concerned. They are excited about their inventions, but they're also deeply concerned about the use of them. So this was my first story that there was something on the horizon where I, you know, they were talking with me about some of the concerns. At that meeting, it was very interesting. I saw some of them and they tend not to be terribly animated when marvelous things happen in technology. And yet I saw some of them chattering about GPT-3, which is the technology that powers chat GPT. And I thought, what is impressing them so much? So that, you know, these luminaries were really marveling over it. So I went back home that was September and let's see, so this was in October. I published my first story based on that tip. So, you know, the reporter's life is to try to stay ahead of the curve so that you all can be informed and get information that you need that, you know, you can use to inform your work. So in October, I wrote this first story, Machines Can Craft Essays, How Should Writing Be Taught Now? And this was an interesting story because I had a little bit of time. I took some time to write this one. I called up experts and researchers who actually had been spending some time with these AI writing tools. This was not a situation where a professor in the classroom found out about this tool and needed to adapt in one day. This was, these were researchers who informed this story and talked to me a lot about both, you know, the, you know, the risks that this posed that, you know, that some teaching and learning practices may need to go by the wayside in the presence of these new tools. And yet there would also be a lot of opportunities in teaching and learning that maybe faculty hadn't thought of before. So I like this story because I liked working on it because I actually had some time to think. OK, we all remember January of this year, right, when ChatGPT was released. And I think I actually am very, you know, when I think about all of my readers or inside higher eds readers, I think about them as real people who are working hard and trying to do well by their students and, you know, with the best that higher ed can be. And I remember being a faculty member in the classroom and having so many demands, faculty meetings and, you know, faculty senate and student and office hours and, you know, grading and, you know, OK, I just worked hard on this lecture and now I need another one. So I remember that life and I remember how busy it is and how demanding it is. And what I think my role is, is that I have a little bit of time to go and talk to the people I need to go and talk to so that you can be informed so that you can do your jobs better. And often what I try to do is look at all sides and lay it out because I know how intelligent you all are and that, you know, you can form your own opinion and many people do land in different places. And, you know, I think that's important to understand that all of those different views. So this story, I understood that something big was happening in January, not unlike what the switch to emergency remote teaching. And when faculty just needed to adapt in a week or two and, you know, they were sent home and, OK, your classes are online now. And something was happening like that in January. So this piece I actually edited, I wrote the intro. But what I did was I, you know, I look for diverse sources in terms of students, administrators and faculty, different ages, different disciplines, you know, as much diversity as I could. And I reached out to these people and I said, tell me one thing, people who had experience with these tools. And I said, tell me one thing if you had to give a piece of advice. And, you know, I put these together so, you know, be deliberate, adjust quickly. Don't abandon pencil and paper question how writing is taught. Think a few years out. I like that one. That was actually interesting in that moment. Delegate, identify shortcomings, you know, so again. So this, you know, reminds students to think another important one, even though these tools were here, they had arrived. There was a lot of advice. And that that piece was I'm really grateful for all of the sources who contributed to it because I believe it helped in the moment. And my job is just to continue following the conversation wherever you all are taking it. So next up was AI writing detection because a lot of people were, you know, there's a lot of discussion about, you know, are these tools going to be used for cheating or, you know, how are we, you know, and or, you know, if the ed tech companies come with their tools, are they going to be used to penalize students inappropriately? You know, there are a lot of questions there. And so I, you know, ended up talking with a lot of mostly faculty for this one, I believe, where, you know, they, there were many who were saying, we need to under, you know, this is not exactly about catching students. Most, in fact, were saying it's not at all about that. But yet there was still value in understanding the difference between machine written prose and human prose. And that not only was that important so that we can understand when something that, you know, a piece of writing that is, you know, putting forth some ideas is written by a human, but that there's actually some really engaging intellectual conversations to be had there about what really does make writing human. So not just the mechanics of it, which I explain a little bit of how the AI writing detectors work in this story, which actually fascinated me. I was really excited to learn about that, their fun words such as perplexity and burstiness. If you haven't learned about this, they're really fun to think about. But also just how can we use this moment to bring out our humanity, which is something to celebrate. Okay, next up, faculty were telling me they were finding that some of them were putting their questions from tests or, you know, you know, essay assignments into chat GPT. And some of them, some of them were saying, oh, no, this is, it didn't do very well. It, you know, would have gotten the CRD or not well at all. But some faculty were telling me I put my questions that I've long used or the kinds of questions that I've often relied on. I put them into chat GPT and they, you know, gave a beautiful answer. So faculty were telling me that they were having to, some of them were finding band-aids that would work this semester, but that they were needing to think a lot more deeply about how to ask questions. And that was actually really compelling to think about. And narrate that change. A lot of the discussion initially in January, February, and let's see this one, you know, I would say all of January and most of February was a lot about something unprecedented is happening. Do we need to worry about cheating or not? What kinds of questions are we asking students? But then I started hearing faculty grow concerned, some not a lot, actually, because, understandably, a lot was happening. But some were telling me that there were some risks that could be posed to students that you could bring this, you know, chat GPT into your classroom and, you know, send the students off to write an assignment with it. And, you know, a student might, for example, ask, is my life worthwhile? Chat GPT is not similar at all. What faculty were learning is that it's not at all similar to the chatbots that are found often on admission, college admission websites or library websites even. Those typically are designed specifically for universities and they answer a very narrow set of questions. They do not go beyond it. They are targeted specifically for, you know, the public, you know, the consumers, the 18 to 22 year olds or whatever demographic your institution serves. And what these faculty members were telling me is that this is a tool from the outside. It was not designed for our specific population and there may be some guard rails needed for these tools. At the same time, you know, there's a lot of, there were a lot of balls in the air that the faculty were trying to keep up because they were also telling me, for example, faculty were beginning to develop some of their own philosophies and I saw at least two camps. One camp was telling me, it's a great tool. Let's keep our oversight and, you know, use it as an opportunity to have more conversations with students and some faculty were saying, I always encourage the students to start with ChatGPT for those students who face a blank page with dread. It can be a really wonderful way to jumpstart some, you know, thought and then the student can take over at some point shortly after using it to start an assignment. Other faculty were saying, that's a terrible idea. Don't do that. That bypasses the students original thought. They need to start with a blank page even if it's hard and they need to see what they think first. So, and then there were some who are neutral and hadn't decided because of, let's all be honest, this is happening very quickly and yet some were also identifying that there were certain populations of students for whom it was really serving well. For example, some faculty shared that some of their more neurodivergent students found it to be a very patient debate debater who, you know, they could ask many, many questions. So, you know, our job is really as a reporter is just to listen to all of you and try to craft that into stories. This story that came out this week by my colleague Liam Knox can turn it in, cure higher eds AI fever. I was working on another story. He picked this one up and I was very grateful is also another story about, OK, an ed tech company has come out with a product and, you know, we only, most of us were only paying attention to chat GPT at the beginning of the semester and already this product is being released and what implications does it have? You know, we're all familiar with plagiarism detectors, but plagiarism detectors actually give evidence when they say there's a piece of sentence or a paragraph or longer that has been plagiarized. They can say, here's a paragraph and go look over there. That is where it was lifted from. AI writing detectors are different. They only speak in likelihoods. There is no evidence and that's a fundamental difference between the writer, the AI, I'm sorry, between plagiarism detectors and writing detectors. That brings you up to the current moment. This one, as I said, was published this week. Let me. So what am I thinking about now? Right now, I'm thinking about this letter that many tech leaders came out with about should we pause AI? And I haven't written the story yet. I need all of you to tell me what it needs to be in it. Does higher ed, is there any piece that we should be pausing right now in terms of these AI writing tools? Because we need more time to draft policies. On the note of time, I want to add that it feels like it's been a long time since January, but I would urge you to remember that we're really in the infancy of this development. There, you know, it can seem like if you go on social media or you hear some of the voices who are leaders and for whom I'm very grateful who are talking a lot about these products that it can seem sometimes like everybody is up to date. And I assure you, actually, that's not the case. There was a study that just came out. I wrote about it in one of my stories that I believe was current as of the first. I'm sorry, it was at some point in March. So very recent, where only 18%, I believe, 17 or 18% of faculty have developed AI writing policies for their classrooms and only 14% of universities have. We're just at the beginning of this. There is, you know, if you're feeling like the trains left the station, it has not, join us, join the conversation. My guess is you're at this conference, most of you are engaged in this conversation and possibly even leading it at your institutions. And that's a really important role, so that's great. But anyway, this is, you know, what I'm thinking about for my next story. You know, given the disruption, should higher ed pause any aspect of AI writing tools in any way? And I've got my email up there. And if you have a dot edu email, I'd love to hear from you. Feel free to send me a short conversational paragraph. A little longer is fine, if that's easier for you. But I'm actually interested in hearing what you have to say. And if you're someone who hasn't been quoted in a newspaper before, I especially want to hear from you. So, but everybody is welcome. So I will leave it there. Thank you, Susan, Peter. All right. Hi, everybody. So I'm Dr. Peter Organichuk. I am a professor at University of Denver, just here in town. I've been at Denver for six years. I still don't understand the weather, so you'd be forgiven for also not understanding it. I described it to a friend yesterday as typically atypical. And I think that describes it well. You never know what you're going to get, other than when there's a conference, it usually snows. So I'm a professor in library and information science. So my role here today is to talk a little bit about the research, the scholarly research side, as well as the teaching side. Sure. So my my research is in text mining, as Cream said, I come from digital humanities and I work in applied methods when it comes to text mining. So I'm very rarely doing algorithmic work, developing the methods. It's more seeing how we can use these methods in libraries and in information science. My work is primarily in two spaces. One is a content based methods for I'll hide that for now. I'll get there. One is content based methods for understanding large digital libraries, right? So a number of you in this room I've interacted with in my past role at the Hattie Truss Digital Library. I work on ways of understanding these large scanned book collections at scales at scales in which we can't really read them, right? My other area of work, a bit more recent, is an automated educational assessment. So the library and information science at University of Denver is in a school of ed. So I work with educational psychologists on developing ways to measure tests of creativity. So creativity is something that we've had tests for since the 50s, but they're very rarely used. They're very rarely seen a test of creativity in education because it's just really tough to grade. You have to have an open-ended test. So the past couple of years I've been working on automated methods for for scoring those types of tests. And I'll show in a moment some some results from from a recent study. Actually, super recent that show the power of large language models and chat GPT included in that space. Finally, I'm also working on I'm currently working on a study of GPT based writing interventions. Essentially, what we're doing is we have a browser based tool that's backed by chat GPT in the back end. And it really helps it works to help students in overcoming the tyranny of the blank page. So it's a it's an auditor. It doesn't write anything for you. Rather, it talks with you to help tease out your thoughts on what you want to write. You know, as somebody that is an immigrant from a non English speaking country, I know what it's like sometimes being unable to having an idea and being unable to express it. So what we're doing in that study is we're comparing this customized discussion around your writing to generic writing advice. In my teaching, I'm also trying to I'm struggling very currently with adapting to the presence of things like chat GPT in the classroom. For example, I'm teaching a program in class now where I'm implementing chat GPT as a tool while trying to contextualize how it can be used. Right. So as Susan mentioned, there are lots of concerns in higher ed around these types of tools and that that really present challenges to how we think about assessment in the classroom. However, library science is a very applied professional. I have it blanked out right now. It's a very it it's a professional degree, right? So chat GPT when it comes to programming is a valuable supplemental tool in programming. The challenge, however, is it really is more beneficial when you know the foundations for what you're doing. So the challenge is trying to communicate to students. We need to learn the foundations because then then you can use a tool like chat GPT more effectively. You know, so if you prematurely use it, you actually won't be able to to work as efficiently. My other role here today is as as a scholar that works in this area is to present some of the jargon. So sort of take us to now when it comes to chat, chat GPT and similar tools. So some of you may have taken classes like text analysis or information retrieval when you're in school. Some of you have it. That's OK. The by historically, one thing that we've worked a lot with is this idea of bag of words where when we try to represent texts computationally, we really treat it as a set of words. And the reason for that is not because it works well, but because it's just really hard to to model how how words interact with each other, like the order, the sequence of words. We've long struggled to come up with ways of using using the information in the sequence of words. Right. So a bag of words just jumbles upwards. Each word is treated similarly. However, over the past 10 years, we've seen sort of a resurgence of neural networks or about 10 years ago, we saw neural networks come back from from their previous heyday in the 90s. And we started experimenting with a type of neural network called a recurrent neural network, which can map a sequence. The problem with text and is there's so much of it. So it was computationally intractable. And the big thing that really made the tools that we're using now possible was this idea called attention. Attention is really just a concept that lets a neural network focus on what matters. So rather than focusing on every word piece by piece and how the dependencies of each word function altogether, it can sort of focus on important words, skipping less important words. And that makes it more computationally intractable. So attention led to models called transformers. And transformer based language models are often called large language models, as we heard. And these large language models, they're built using really large training corpora. So the other thing that really became prominent 10 years ago was this idea of transfer learning. So rather than every time you have a task, you take a corpus of text and you teach, you create a model yourself off of that corpus. With transfer learning, the idea was you repurpose other people's models. So starting with a tool called word2vec exactly 10 years ago, we really started working off this idea of somebody with a lot of computational power trains a really powerful model, and then we can repurpose that model and we can fine tune it. So if we have a very customized task, we can train it a little bit more based off of our specific use case. In contrast to defined tuning, where we tailor a model to our use case, the other thing we've seen in recent years is the remarkable effectiveness of few shot or zero shot learning where you give a model very few examples and it understands your task, mainly because it learned the language so well that it can interpret it. So that's what we're often doing with something like ChatGPT. We're using prompt-based, or we're using prompt-based few shot or zero shot learning. So zero shot, you don't give any examples, few shot, you only give a few examples. But you're using an out-of-the-box model in any case. A couple other things that bring us to now. One has been the idea of scaling laws. So there's a paper that now we're starting to see pushback against that came out of OpenAI a couple of years ago that said, the bigger your model is, the smarter it gets. So that's why we've seen OpenAI training these bigger and bigger models. GPT-3 cost $12 million in computational training. Resources to train, for example. The other thing we're seeing more recently is what you're training off of matters. So you want a lot of texts, so usually you use crawls of the internet, but you also want it to be clean, so deduplicated and without a lot of junk documents. The last thing we've seen in recent years has been a concept called reinforcement learning from human feedback. And that's really what that allows us to help guide the models from human feedback, exactly what it sounds like. And that gives us the ability to tune these models a little bit better than in the past. So we can introduce more safety and anti-bias interventions. And that's how chat GPT or GPT-4, Google's barred, they work in that way. All right, and then one last little thing is how these models actually work, which is they're probabilistic. They're trying to, based on an input, they're text-to-text generative models. They try to generate an output based on an input, but they usually have a temperature setting where there's some randomness thrown in. You can turn that down, so you get the most likely answer. But that randomness actually, I would argue, adds to a lot of that human-like feeling, the fact that it doesn't always give you the robotic best answer. All right, so I wanna show you, so that slide I had hidden was just some of those definitions, but there you go. So I wanna show you results from a recent paper on scoring originality, which is a part of creativity. So up until now, the best automated scoring methods have operated, let me show you that column where it says overall, they've had a correlation with human judges of 0.12 or up to 0.26. Not great, however, over the course of a larger test, that starts looking better and better. In our recent work, we used large language models, I hope that that's a little bit off, we used large language models that were fine-tuned, and you can see it's a massive, massive improvement where one is perfect correlation with human-radar, zero is no correlation. We get up to 0.82 correlation, right? So these models are able to learn a bit of that sense of the language and that sense of the task, and I won't go into the details of the task, but what matters is it's really big. And it also learns what this chart in the corner shows, it also learns very well with just a few texts. All right, in contrast to fine-tuned models, you can also do the sort of prompt-based models. Where you actually give some examples to the model and it tries to guess based off of those examples, it tries to guess the answer for other examples. And you can see if you give it zero examples, GPT-3 does fairly poorly, comparable to the previous state of the art. If you give it five examples, it already surpasses that. But not as good as fine-tuning. However, adding chat GPT in there, it's a little bit better and much cheaper. And then if you add chat GPT-4, you can see now we're approaching what fine-tuned models do. So this is using just prompting. So, and again, for reference, that's the baseline there. So, one thing Susan mentioned was faculty mentioning, this didn't give a very good response in some cases, while sometimes it does. That sort of concern, I think, is limited. So, or is time limited. So, one thing that I think is important to know is the quality of these results just keeps getting stronger and stronger as the models get larger. And that will factor into any conversation, I think, moving forward. But yeah, with that, I yield my time. Thank you. Thank you, Peter. Hello, everyone. I'm Brie John. I'm the Natural Language Processing Specialist Librarian from University of Florida. My department is a new department too. As you'll hear, my title is pretty new here. Is Academic Research Consulting and Services Department. I'm very glad to be here today. So, how many of you have used chat GPT so far? Very nice. Okay, this is a nice crowd. Yes, yes. Because Karim mentioned that I might be the first AI librarian get hired, I would love to start my talk with, introduce you a little bit about what I do in my libraries. So here's a little bit about my training background. I got my BA in, oh, actually it's a BS, sorry, it's a typo, in Educational Technology, where I learned pedagogical theories and a little bit programming from my bachelor degree. And I did my MA and PhD in major in linguistics and minor computer science at the University of Minnesota, Twin Cities. And what my project was about using NLP AI machine learning method to document in dendro languages and it's also sometimes called low resource languages. Today, why we're so fascinated about chat GPT is because we all know about English. And if you use another language to test it out, especially the low resource languages, you will find this is rather not very useful too for your research. So these are some of the language that I've been working on before. So when I introduced my role to other people, I love to introduce NLP as a combination of computer science and linguistics. And I will put an emphasize on the linguistics side, no matter the overload information you got these days about chat GPT or any natural language processing related application, I think they're all trying to figure out either all of these aspects in linguistics or some of them based on the acoustic patterns, how the morphemes working together, morphemes means part of the words, maybe not the words. When morphine put together, we got a words. When words put together, we got a sentence and the sentence has structures. When sentence has structures, we start interpreting the semantics meaning of it. And once we have semantics about it and you put people together and you share your thoughts and that's pragmatics. So usually NLP is trying to tackle all these questions and I would rather say those are traditional questions that we're still working on and AI machine learning in general is trying to help us to take another advanced step to solve these traditional questions. And here is some of my department structure. You might be interested. We have a lot of folks that expertise in other area and I happen to be the first little icon named artificial intelligence. So if we got a patron, question, consultation or collaboration request and they will see that and they'll click on it and then find my contact information from that. But we do have all sorts of other services that my department provides. So when I introduce myself to my patrons, I usually tell them I have two hats one is AI related where I support all research that done by student groups, faculty or labs, university staffs on their AI or NLP related projects. And I also conduct internal library employee trainings to just learn a little bit more about AI and we tend to make it a little bit low mathematic low programming to just let people get interested in this area and not scare them away from the programming side. And I also collaborate with different research group. If I see, I have a pretty useful role in their projects. I will collaborate. So later I will share a little bit about what are the ongoing projects I've been collaborating on campus. So why I got hired? Why University of Florida? We have this pretty big AI initiatives that there are many, many, many implementation about this initiative. And I'm just highlighting this two direct influence me or my department. One is that we have campus wide AI faculty hiring. So far there are more than 100 faculty members that doing AI already hired and started their working. And the goal is to implement AI machine learning into all departments curriculum. And so I was having to be the first one that hired by the libraries. And I also, my daily job is to first log in to this computing system. We call it HyperGator. It's partially sponsored by this chip company called NVIDIA. We have some allocation that granted to my department and we have a group of people support unfunded students project that if they're, because it's not free. If their faculty member cannot afford it, but we also do not want this encourage students to give up their project, they will come to find us and we give them a short period of time to test out what they can do with this computing cluster. Okay, and some ongoing projects. So this first one, I have three cases here, but I would like to just give you a type of what kind of collaboration I usually do. So this is a clinical research lab. So it's a lab that involves multiple faculty member and the graduate students. So their mission is to study clinical notes. Sometimes we also call it EHR, Electronic Health Records. And there are notes by doctors or nurses, it's very messy. And their goal is to use NLP to extract information that this specific group people care about. So they study infant feeding status. So it's a pretty straightforward pipeline and you find your way to manually label those relevant information through massive passage and you find the relevant, you just basically test out all these NLP related algorithms and implement them and see which one can help you predict the best labels for this information. So we test it out and then find some really great algorithm to suggest maybe this in the future other labs or other university or research group can keep using that. And here's just an example showing you different hospitals, groups that have very different forms to keep patient information. And it's really a time consuming task for new doctors and new nurses to go through all these and find information. And the worst case is there are very critique diagnosis but overlooked by doctors and maybe that's the cancerous. And when they realize, oh, that's a piece of information that's already past the best of treatment time. So doing NLP for EHR data is one of the leading research in NLP field. And so how we do that as a group, there are many, many annotation tools that different labs would like to do. Usually this is just an example, it's called Team Ted but it's just an example, there are many, many more. I believe that there are publisher companies also offer some annotation too and we would love to learn about that. Usually these tools will give you some agenda that a PI can put a very structured nose and assign to their group of people. And also see, okay, this person think the most important information is this word and this sentences but when the same passage pass to another person in the lab and they have a different opinion and the PI will get a flag, okay, different people disagree with this. So this is really fundamentally reduced bias in AI when you have multiple people to judge the same piece of information. This is just a screenshot, sorry about this, words is very little but this is just very useful tool so you don't have to program that much. You just highlight the information you need and then select the labels that you think is correct. If there are disagreement between annotators, the PI will get flagged. So that was a collaboration that I'm currently doing with a lab at College of Medicine and this second one is a pure faculty group. They're from agriculture and they wanted to use NLP to explore urban green space planning. So why NLP is relevant to this to agriculture. They wanna know users' opinion about how to plan a city urban area more friendly to users. So they are actually targeting on Google reviews, YOP reviews and trying to see what comments that people, users, park visitors care about. So that's where NLP can get in and there's an NLP task called topic modeling. So it just review all these massive messages and it suggests you a certain number of topics that you might wanna start with. Yes, there are biases everywhere and you might miss some really great topics that the model didn't return to you. That's why every time when we say we apply machine learning method is just to begin with. There are many, many other follow-up steps that needed before the final product. And the third one is a pure student group and they are, I work with them. This is a group that I code more for because they're new. They come to the library and find me. They say, okay, this is what we found. We're all medical doctor students and we're fascinated by from, so there is a database called PubMed if you're in a medical area, that's the first place you go. And so they are, there are some key words about this group. They care, they are the future surgeons and they will be doing surgery for pediatric, this area, the second key words. But they often find that they wanted to find the basic research literature. But often the PubMed returned them a whole bunch of paper that is clinical based. It's not very useful. So they come to me saying, okay, do you think NLP can play a role here? Maybe you can do some supervised learning. Well, they did have some intro of machine learning. That's why they found me. But they did not have programming backgrounds. So I helped them use this large language models to tokenize their giant literature collection and then further developed this classifier that can predict either a literature is more about pediatric basics research or clinical. And there's just some results that is not very important to this meeting, but just I would like to share with you, these are all based on large language models. And actually we heard about large language models a lot. And how about we think a reverse way? Is there any small language models? Why people don't talk about small language models? Why all of a sudden large language models? That's a really fair question. It's just because large language models, I guess we have to thank chat GPT really talk about, we know what language is, we know what models can be, but language model, what that is. So chat GPT is a really good example. Thanks to the media. Let us learn a little bit about language model. The traditional small language model really means that it cannot do much things. You separately train a thing with rule based, right? You say, oh, if this, then that, if this, then that, but you cannot adapt this to another field. Large language models, yes, is mostly deep learning, neural network based and it can do a lot of things. One of the important things is today, a lot of AI featured things are powered by large language model as a core. So it cannot, usually cannot stand by itself. It needs a lot of downstream specific tasks that multiple people, folks today have mentioned. Fine tuning is a very important step that you use this gigantic language model to build something specifically for yourself. And that is very doable because you don't need a lot of computing resources for a fine tuning project because presumably your data set is relatively small. But the outcome is just very great compared with those earlier small language models. So apparently there are history about language models. I think it was starting in the 1950s. Georgetown experiment, if you have heard about that. That's a task by IBM Watson. They're trying to translate 60 Russian sentence into English and that's how where language model started. If you're wondering about that. So here are, so how about chat GBT today? Chat GBT definitely made my work a little bit more busier these days and I got a lot of questions from patrons and some faculty members said, okay, I need to tell my students. I'm teaching this computer science course. Tell me more about this. I need to teach this chapter. And so they ask about what reinforcement learning is. So I will say one of the biggest thing about chat GBT, very wonderful is about this reinforcement learning. Basically, it's just review. We all can be this in part of this reinforcement learning because once you use chat GBT, there is a thumbs up and thumbs down button. And that's reinforcement learning. If you press up and they will add a little bit weight, more weight to say, oh, that's a good answer. My customer love it. If you thumbs down, you contribute another data set for their entire training process. But before they launched that, who would they use? And they hired a foreign country, very cheap laborers in Kenya by paying them $2 per day or something or per hour, well, I think it's per day. And it's really cheap labor and I would like to bring some concerns about you know, it's not building fancy large language models it's not only about the programmers it's also about the human loop. We have to value the intelligence that other people bring to the team. So that's about reinforcement learning and other popular questions. So what's the goal of chat GBT? What do you want to do? I think chat GBT is really just want to have a conversation. It's not originally aiming to give you any evidence based thing for your professional work. If you think that's a pretty good, you can have a pretty good conversation with this. That's their original purpose but someone could come up to me and say, wait a minute, I don't want to have a wonderful chat casually, I just want to have something help my work. So we found out the chat GBT lies a lot of time about citations, reference information but if you think about in the engineering side it lies but it lies in the right category. It lies in citation. It didn't lie by giving you an image of hot dogs, right? You won't feel threatened if it returned you a picture of a hot dog but because it's so close it's trying to keep up the conversation but because it's statistic based there's no fact checking well established in this system. So we found, okay, you're trying to give me a reference but it's not existing. Again, that's not their original goal but it could be their immediate goal. Just because the work pipeline seems pretty straightforward if someone want to build a fact checking system to against it, right? Got a database of all the real ones. Chat GBT is here, let's go checking, right? It's pretty straightforward. Someone has to do that. Yeah, any unexpected outcome, of course. I'm not gonna say much about it. So this is my last slide. I wanna leave enough time for, oh my gosh nurse. Okay, my goal as a new librarian is there are a lot of needs to just learn basic things about AI, about natural language processing in general. So I love how media bring chat GBT to everybody's attention. So I will have opportunity to say, okay, chat GBT is just one example of large language models. Large language models is just one example of language models. And language models can do a lot of different natural language processing tasks. So depending on your own research paper and the data sets that you have you can do many, many more. So I'm back engineering everything and trying to bring more AI librarian to my circle. Thank you so much. Thank you for all and all. Actually, I'm gonna skip my presentation and give you, I'm gonna do a special Zoom for that. I have to make an executive decision because we barely have three, four minutes. So I'm gonna do a special Zoom around the geopolitics and the economic model behind those companies because there's a whole thing behind the scenes. And also this AGI, Artificial Intelligence Sentience and the divide between researcher whether we should stop or not. And even the Godfathers of AI are divided. So I'll do a special session for that. I'm gonna open the debate now. Like, please ask us questions. We need a volunteer. Otherwise we can close. Oh, okay, yes. Clem Guthrie University Librarian and University of Hawaii. Also the interim director of the UH Press. How do we deal with the ethical issues of those false citations? So currently I have a researcher in Canada who contacted the press saying the press published a book which the press did not publish. She got the citation from chat GPT. She's now being accused of academic dishonesty. Wow, you wanna answer? Yeah. The short answer is I would report on this. I think it's the question is who's doing the accusation and the policies are being developed right now. And if the policy was not in place then that would be a concern. So this is all being worked out. I think that these conversations, I'm having a month from now and I'm essentially leading the Digital University's US Conference in Chicago. And every session I'm leading, I'm gonna be talking about ethics. The ethics of going to college in the metaverse. Ethics of AI in research, in publishing. So I don't have an answer. I don't know, do any of you have an answer? All I can say is that... This is work. I think at some point we really need to think about the literacy skills around tools like this because they're here and they're good for some things. They're very poor at other things. How do we equip people to understand this will hallucinate citations? And I have to say, from the perspective of an educator, I have no idea how you teach that literacy. Like Brie said, it's almost... I would liken it to understanding Wikipedia when that came around. Wikipedia is so often reliable except when it's not. So you get sort of lulled into this false sense of trusting it. And I think a tool like Jatt GPT is similar in that it feels right, it feels right. And like Brie said, it'll give you those fake answers in a realistic sounding way which makes interpreting and judging the quality of that very difficult. So I don't know where we go in educating people but that will be, I think that'll be a big thing in higher ed. And I would just add quickly that these tools don't exist only in academe. I think you were hinting at that, that the students are going to graduate into workplaces where they are omnipresent. I encourage all of you to take the lead because if not, the ed tech companies may. Last question. We'll have to arm wrestle for it. All right. Rock, paper, scissors. It's all good. I just was wondering, and sorry. The gentleman from Denver, what are you doing for your students now to prepare them for what happens? Because I'm gonna be probably hiring some of those kids. And I'm just wondering, are you doing prompt engineering? What kind of, how is it impacting the next generation of librarians? Yes. The short answer is, I'm not sure yet. I'm sort of building the car as I go along, like many other educators. So right now, I'm teaching a class that's really data analysis. And what I'm trying to do is establish, here are these fundamental skills. And then what I have planned for later in the quarter is a session on sort of bootstrapping working with ChatGPD as a pair programmer to bootstrap your skills to be able to do something just one step further than you were able to do it. But how is it still up in the air? And it's really, really a new consideration that many of us are just starting to struggle with. I would just add a very short answer, I think. It's better for students to get their hands-on experience as soon as possible. Find a very small-scaled first experiment to let them get started from data cleaning. ChatGPD or GPT-4 spend, if not more than 85% of the time just dealing with data. And they should get their practice started as soon as possible by working with real-world data. It's a messy and a lot of learning process. Thank you. One last question, then we close. I promise we can keep this super quick. Yeah, yeah. If you want. I'm really interested in your panelists, how you would answer Susan's questions. If you want to do thumbs up, thumbs down, would you pause AI right now for six months? AI research. Don't pause. Let me tell you this. Like, it's really, they know it doesn't work. Like, if you follow all social media, how they're talking about it. But it's kind of just signaling we need to do something about it. People are saying, okay, if we do, like, how about the other companies? How about China? Like, it's not gonna happen, that's for sure. But there is huge debate, and the media is covering it. So, slowing down, and some like, yeah, like Ken, he's also one of the founders of AI and the way, and deep learning. He's saying, like, there's nothing there. Why we're freaking out? Like, but some really, really AI thinkers, we're doing this, and there will be a harm. And when you do like surveys and scientific survey, literally split, 50, 50, between your average Joe, or really, really AI scientist, AI expert. Totally split. This is why it's hard, and actually, Susan is gathering feedback to write an article. So please kind of, you know, give your feedback. I would, I'd say the arms race to make bigger and bigger models, sure, pause that. I think it's really important to concentrate on bias reducing things, right? Because these models are learning from human texts. So, from human generated, you know, things that we wrote, which means they have the same biases that we have. And if we're constructing something, we really should be able to do better, because we know people have all sorts of biases, in terms of gender, for example. So I think there needs to be more work. The models have gotten strong. I think the research on tempering them and doing something better than just recreating what, how people write should, we need research to continue in that area. And hopefully it does. Yeah, I also agree. Just to put another more optimistic side. A lot of us, I believe, are not very satisfied with a lot of repetitive work tasks. And if we can free some time by using very good tool, unless it's last bias, then we can free some time to do something more creative and meaningful. So it's eventually trying to make life easier, but when something started, just got started, it can be a chaos. As a journalist, my contract, it's actually tradition and journalism, but also specifically my contract says that in a public setting, I'm not supposed to take, you know, make my opinion clear, because that could potentially alienate sources. So I want to hear from everybody. Okay, thank you all. Round of applause for all of you.