 Welcome, everyone, to the first AES New Zealand seminar of 2024, Artificial Intelligence and Evaluation by Dr. David Federman. This seminar is recorded and it will be uploaded into the AES YouTube channel. I am Marini Sankar, the co-convener of AES Aotearoa, New Zealand. And before we begin, I'd like to acknowledge the traditional custodians of the diverse lands in which we all come from. I'm speaking from Poneke, Wellington, Aotearoa, and I acknowledge the leaders past, present and emerging from all our lands. Just now to please keep your mics on mute and type in your questions into the chat. We will have about 15 minutes at the end for Q&A and I apologize in advance for any Wi-Fi interruptions that might happen during the session. I'm pleased to introduce Dr. David Federman, founder of Federman & Associates, renowned for developing the empowerment evaluation approach. David has worked in diverse contexts and countries, including Japan, Brazil, Ethiopia, and Aotearoa. Former president of the American Evaluation Association, David has earned several prestigious awards and he was named the top anthropologist of the decade of 2020, which was celebrated in Times Square. I'm so excited to delve into AI. So over to you, David. Thanks so much. I appreciate it. And thanks everybody. Nice to see folks I haven't seen in a while. A lot of friends, a lot of colleagues in New Zealand and in Australia, of course. So thanks so much for having me today. I'm going to go relatively quickly because we have a short period of time and I'll already let you know about an hour or two ago. I was on the net to make sure I'm up to date because it changes that quickly. I'll be wrong on a number of things in about three hours. So hang in there. Things are definitely changing rapidly, but the core should be pretty solid as far as what you hear today. And when I, all of the pictures you see are AI generated using dolly three, just so you know, they're not just random shots. And take your time and you'll see I put the text, the prompt I used to get the picture underneath most of the pictures. So you get just a feel for how precise you have to be in your language when you're prompting things to get what you want. Think of it like a Google search. It's not quite right the first time, second time. Same thing with all of AI at this point. Better and better and better, but I, I, you know, want to warn you, some of it takes a little tweaking to say the least we're still, it's like a baby. We're just, you know, helping to nurture it and develop it giving it feedback and it's changing very, very rapidly as you'll see. Let me keep on going forward. This is just the text I put in to get that picture, which is not entirely what I wanted but close enough in terms of artificial intelligence. On one side holding the scales of justice, because this is true fair fairness equity that we'll talk about. And then there's the whole concept of, and then you know, an evaluation also being sort of looking at for accountability and that sort of thing. And then you have the other side, this young girl holding the fish and that sort of thing, and maybe helping someone learn how to fish, which is more the collaborative participatory apparently kind of thing. All in one slide, just playing around with. Well, this, the reason sort of that I'm doing all this today, at least in part is because we introduced this session at the American evaluation association meetings in October 2023. And you can see what kind of response we got we had no idea was going to be this well represented it packed the room, people on the floor on the hallways are coming through the crowding in and that wasn't even everybody it was amazing. So, based on that, we thought, well, we better keep on learning and sharing and that's all what none of us know everything and certainly not I don't either. It's a matter of us all learning together and that's why I'm here today as a consequence of this session that was so popular so you know well attended. We created a LinkedIn page, I feel free to join and join in and if you're got some interesting insights, put it on there for all as I say seriously learning this together. We also as a special edition of new directions that came out evaluation artificial intelligence, wonderful resource as well. You can see this just blew up. I mean, you know, from even our work just in October. It's gotten to be this amazing proliferation of activities and discussions and dialogues, and we'll go over as much as we can today in the short period of time. So let me give you just a very brief history of a feel for this for those very new to this area in the 50s and 56 period. That's really the birth of AI. It was at a Dartmouth kind of workshop. And if you're familiar with Alan Turing that's a lot of movies about him. He really did the undetected imitation test where you ask a computer something and they respond, and you can't tell the difference that it's actually a computer. The 56 to 90 period. That was what we call the AI winters. It really was, I know, surprising, but declining funding, very little interest. The only thing that was a major kind of interesting that people's attention was they developed a chess playing computer system that defeated human champions to give you a metaphor for the differences in the 50 to 56 period they actually came up with a checkers playing computer. And the sophistication is a whole different level and you get to the 56 and the 90s where it was literally a chess playing computer, but it gives you an idea as a metaphor of how much it advanced in terms of analysis. 2010 to the present represents the age of AI. Really, in terms of facial recognition, self driving cars, that sort of thing. That's in a nutshell, I'm going broad strokes, of course, they have a feel for the pretty rapid development, but believe it or not, it's not a straight linear line. It was really of no interest to say at least for a while. And then boom. Once again, the text on the side not critical, but just to show you what I use to come up with some of this picture of the main figures in the development of artificial intelligence. The reason I highlight this is not just to remind you of the different kinds of prompts you use, but when I put that in before one of the pictures had mostly all men. So you have to watch for the bias which we'll get to as well. That's built into the internet that feeds. Yeah, but you can get around it by putting a check on it which will we'll talk about. So, I'm going to just give you a very brief introduction to the AI alphabet soup I call it. Don't worry about this, but it's just we have a feel for the language and the terms that are used. It'll give you a better feel for when we get into the core in a minute. Artificial intelligence is in computer science, it really it creates machines to perform tasks requiring human intelligence. There's two basic approaches which is the narrow AI place chess driving car that sort of thing generative, which is more reasoning learning and creativity. And I put those in pictures on the left as you can see draw a computer playing chess and self driving car on the top on the left. On the bottom, a picture of Einstein formulas light bulbs and old red schoolhouse and a question mark. You see how concrete you have to be in some of the prompting to get what you want and you'll see if you just put in something, you know, general like give me reasoning and learning. You're not going to get anything meaningful. It's not able to do that into the pictures as well as we'd like at this point. That's why I'm trying to show you how concrete you have to be. Let's continue with the alphabet soup because a lot of little terms. Don't worry about these, please, but I just want you to get a feel for it. You hear the language helps you speak in the logic of this. That's all neural networks. That's the core of what we're talking about when thinking about AI. They're like biological neurons, they're layers of interconnected nodes, final outputs classification prediction generation that sort of thing. The bottom line of the neural networks is, you can have this supervised learning, which is input output. In other words, if I put in two wheels, and I see it's blue and it's got handles. I already have to put in what that's going to be in my list, which will be a bite. So it's input. This description will lead unsupervised learning and that's what we're really interested in today. Is it learns and it discovers? It just learns to discover how to find it. You put in that description. It finds it out there as what is the most common kind of con of conception that fits what you're describing. And that's, don't worry, I'll go into more detail of that in a minute. The first reinforced learning is critical to all of us as evaluators. That's feedback and it learns to optimize. So in other words, you're letting it know it's working or you're letting us not working or that it's biased, etc. And it feeds it back into the system and you'll see how incredibly important that is for all of us as we're playing around with AI. So a picture of neural networks there just to give an idea of what it might look like when you have to start using that, which we'll talk about shortly when we get to Dolly three different software. Let's be just do a tiny bit more on the soup and we'll be finished artificial intelligence. There are large language models that call it LLMs. So type of neural network train on a massive text like the Internet, tons of books, you name it. Generative kind of natural language increase is what it's able to do. It can learn language skills in self supervised manner. And that contrasts with the traditional language models that you rely on rules and invitations that sort of thing. Last one, you're going to hear API is a lot. That means application program interface is just the way to connect to when you use GT for example, or HTTP when you want to connect to this massive data. So you can't just do it. You have to have some interface. That's that's all it is. So this is not a big deal. I'm just going to give you some language to play around with so you're not like thrown off when you do more digging into this than than today's session. So how the heck does this thing work right artificial intelligence. It combines massive amounts of data with sophisticated algorithms to learn and perform tasks in a way that mimics human intelligence. And watch this, this is going to sound very familiar with what we do as evaluators data collection and preparation. This is an algorithm so we have tools to analyze things right and to sort them selection and training training. This means you have a massive data set that you're going to be playing with to figure out, you know, how what the right responses that which you'll see in a minute. As we go through this, you do evaluation of how well that works, and you refine it. And then you have the application and usage that's all this processes. Once again, you can see I use draw a picture of machine guessing at the next most likely word in the sentence, just to show you what this might look as a visual and what it takes to get that visual. As you play around with this you'll see you have to play around with fair amount to get what you're looking for we get to the visuals. This is the heart of really what we have to say today about the logic of how AI works. It's all simple probabilities. I know this is amazingly complex and sophisticated, but on one level on another level, it's not all it is is math and probabilities that's all this is. It's calculating based on all the text that you put in there all the references everything else, what's the most likely letter to come next after after that first level, and the second letter after that and the third and the same thing for words. Watch this, you put in United States of what not pizza. America. I know is the most likely term that's going to come after that phrase. That's really what this large language model stuff is all about. I know this really does. That's how simple this whole thing is on a certain level. When you think about what is AI. So don't worry about all the alphabet soup don't worry about all the complications. It's really based on probabilities. That should shape all of our thinking about the power and how useful this is and how also misleading it can be at the same time. I said draw a painting of a machine running after the word America in this one to get that image applications let's get to the heart of today. That was just good background I think solid give you feel for really the thinking behind what AI is about. You can write letters, blogs, social media posts reports summarize articles tons of our Google will give you like a list of articles to then go and read and find this finds them summarizes it and presented. It's a whole big game changer as you already know. Draw pictures like the one now all the pictures once again that I have here are from AI systems dolly three in particular. You can outline class lessons when you're teaching and we'll talk a little tiny bit about teaching we don't have a lot of time but a little bit about why this is a problem for some but not shouldn't be for others as we learn to really change how we teach. You can write code because most of it's code right so it does that very well. It can analyze documents and pictures. And I want to highlight this has been amazing I do a lot of work in health and in medicine as you probably already know. I want to show you something very cool on diagnostic activity including medical x-rays watch this. Oh, this is of course artificial intelligence as you can see looking at x-ray. What's this. This is a trip. This is going to be this amazing. As you can see over here. AI has access to in this case. I'm going to show you today over 100,000 images of retinas. That's the one in the middle. That's actually mine. It enables ophthalmologist to make more precise diagnosis of diabetes blood pressure kidney disease Parkinson's liver and gallbladder disease heart calcium score which is used to test if you may have a heart attack. It's likely and Alzheimer's AI is access to colonoscopy images. That's in the far right to identify polish better than gastroenterologist. And then the left AI access to x-rays diagnostic accuracy surpasses surpasses season radiologists. You don't get tired at the end of the day like a radiologist does or gastroenterologist gets by the end of the day. It's consistent and has a massive data source. It's incredible. If you want to see more about this, look at the TED talk by Eric Topol can AI catch what doctors miss. Already the precision. This is only been around for what a year and for most of us really since November, but I think open AI opened up in about 2022 by the call. In any case, look at that those TED talks if you want to see more details than I can give today at the and this is can you imagine what it can do when it starts to see a million images. It's going to be around 200,000 which is of course could be done probably in six months not in the year this will all be. It's amazing how fast as we all know this is changing. Anyway, let me keep on going. Problems, there are problems, as we all know, hallucinations we've heard a lot about hallucinations, hallucinations are not lies. It's stretching knowledge and it's not malicious and I'll give you an example of what I'm talking about in a second when we talk about the moon. Confidentially, they're issued a proprietary information that is a learning system that it can then take as the wealth picks up. However, you can now already get around that by having personal language models on your own part of the graph. We'll talk about that in a. Attribution so Wild West is how do you respond to whether it's a real reference or not. Well already, they know our concerns and complaints and spend feeding back into it. And the first stage is to insert your own references and prompts and ask for references, but it can do more than that already and that's why I'm glad I checked about two hours ago to see if that's even updated, which it has. Sources of data as per sources triangulate bias is absolutely biased and gender race socioeconomics and their significant. You asked for correction you asked for feedback. It's now doing it for itself already in response to our concerns about these issues not perfectly, but already in that emotion as I'll show you in a couple more slides. Employment people worry oh I'm gonna lose my job. Well, if you're doing something repetitive and at a lower level, and you're doing simplistic kind of programming, for example, yeah, you will there's no question, which may allow you to operate at a higher level than you're operating now if that's if you're in that situation. But it isn't. It's like you know when they had their rotary telephone folks. They were displaced. The question had to learn new skills when we had when we transformed the whole system of communication. Equity. This is a really big one. The ag gap is a big deal. And if you haven't seen the term the mode. I'll show you what that is in another picture, but it's really when one agency like Apple or like Amazon or anybody has a competitive advantage over other companies. They can protect themselves and interest themselves. And then they have a priority that transcends what you know we could ever compete with the democrat democratization of the net is really about about AI sorry is really about making AI available to all of us. It's so that no one had that massive competitive advantage for us as well. Let me show what that looks like in a sec. As I say I mentioned the hallucinations first. Here's an example just to understand what close nations really mean versus the hype about what's lying to us and other stuff. When you ask AI to show us all parts of the moon all sides. It shows you a picture that's not true. It's partly correct and part of it is made up. It's not lying. We don't have a picture of every side of the moon so what it does is it's trying to respond to what you're asking for and it fills in the gaps by interpolating the differences and the links between the points it does have. It tells us what we want to hear. I think that's all wonderful and perfect by a long stretch of what we need for accuracy, but it's important to understand that rather than just thinking it's lying and trying to fool us and whatever. That's what this hallucination is really all about. Briefly I just want to touch on these if I could with equity it's a big deal we want to look at issues of race, gender, religion, because what's in the net for example as our data source if that's what we're relying on and we're changing rapidly to our enterprise sites that are protected from going on to the net and you can control the quality of the data right now we're talking about uncontrolled quality of data on the net. And of course it's going to be racist going to have gender issues going to have religious bias of course it's reflecting what's in there. So that's what we have to talk about when we talk about what's the algorithm, what's the databases being trained on etc, which now can be controlled to be of higher quality than what we're using at the moment. And as I just mentioned before, it's really the difference between in this case, we talk about democratization, the big AI, you know, companies that are controlling that or have sort of this, we would call this way or the power associated with it. And the technology is now the more it's getting smaller and smaller in terms of what other companies can do what we can do etc. And I think it was Google was the first one had a leaked memo about, we don't have a mode and that's what all companies want to have and luckily they don't, because it allows us to have a greater chance at democratization of this that we don't have that gap without big mode. I'm, I'm a professor but I'm still a student I go I've been to a number of courses on AI as well. And one of them was done by the night foundation I did a valuation of them many years ago there for journalists. And they invited me to their sessions. And one of the things I posted in my assignments was that we have to look at piercing the AI Dale and that's I drew that or they drew it by some of my words over here. What's behind the algorithm what's behind the data is being trained on its exposing explain the logic AI role in government business and academia. It's the same as reporting conflict of interest bias etc. You know where the money coming from. I've read a number of studies which I can share with you we can look on the net for rigor is important because you get false positives is a famous case with the shops. For example, person landed in jail in jail for a year with false data about those AI generated facial recognition software creates distortion very often with ethnic minorities if you ever try to you'll see what I'm talking about. Is it improving literally day by day. Yes, obviously, but only because we're feeding back information that doesn't quite right we're asking enough questions that's realizing it's not producing the right thing etc. The point is, our job in part is to use this to enhance our work to make it easier, but it's also to look at behind the veil. What's going on when we need to. Let me mention some interim solutions and they are changing to bear with me, literally as we speak but some of them are things like this. I carry information we don't want your private information going on the net for others to, you know, learn from or steal or whatever, they have personalized language models already for only $100 to $200 you can train personalized language models on your iPhone and home computer already. So you can control where there has access was out there in the net for others to steal or use or whatever attribution and accuracy, you can add references or ask references. Here's an example of what's going on already that's already responded to this concern. It's amazing. And racial equity buys the same thing you have to search for bias report it provide feedback. If you use vetted databases which is what I'm talking about these sort of either enterprise models or proprietary models. You can vet that data so it's quality data before it's used for analysis etc. This is like in real time. Okay, this is like, like a day or two ago. This is copilot, which I'm using a lot now. And I just said, you know, something about describing API is and that sort of thing. And guess what, I didn't even do anything like I normally do, which is please ask for references. I didn't do any of that. I just asked about API is and helping to explain what that is right. It put a hypertext link if you want to learn more about how API is worth. And I clicked on that and look at it shows you the references. If that's not enough, you've got the 12345 footnotes you go to as well. I didn't ask it for, you know, the veracity of the information it plunked it right in there. So they're self correcting things that are happening as literally we're speaking right now, providing references automatically like this one in copilot. I didn't ask it, providing additional web references I didn't ask it. Providing footnotes, it did it. The point is a lot of this is also minimizing costs is reducing entry barriers when you can first reduce it to, you know, free which we'll get to in a second and reducing hate speech also because it's constantly checking on itself in that regard. That's why this is really important because it's a learning model, but it depends on us asking the right questions and then asking it to check on itself, which it does. Sorry to put so much time into that part I want to go now into the chat boxes or making a transition here. So you have very concrete examples of what things can what you can play with. Once again, this is 100% correct. As of right this moment and then of course I'll be wrong in about two or three hours so hang in there with not probably three or four weeks, I could be wrong. I would prioritize. Bart is one chat box is really cool done by Google. And when I say soon by Gemini it's already happening like literally right now. And that's what's being Gemini pro is actually what's powering it so it's making it much more useful than it was. And that's the bottom of my list as far as how well these perform, and I know I'll be 100% incorrect in a year because it has access to everything Google does. It just doesn't work well yet, but it probably will surpass everyone. One of the most common ones is chat GPT 3.4. It's free. The problem is most of the data is limited to 2022 they're adding to it slowly and surely but it's really do you want to know something like what's the best you know, T pot to buy today it won't be able to answer that question correctly. It's nice. It's less. Let's see, I would say it's less accurate but faster than chat GPT, which is the most popular one of course out there, but it costs $20 a month. Is it worth it. Yeah, it's sort of. But if you ask me no, I'll tell you why a second. But it's a very powerful it's the best one out there. Accuracy in terms of what we're looking for. And it's the learning rate is incredible for itself. Quad is another one that handles long texts better than most other ones so depends on what you want to use this for having said all that. If you want my recommendation and I have no vested interest in any of these whatsoever. It's definitely at the moment, Microsoft. Why, because it gives you chat GPT for and Dolly is a it's way to create image images all the pictures you've seen I use Dolly for that. Dolly three for free. So I mean, you know, you use edge to use it. So from my thinking, it's, it's the one to go with at the moment. If you want to play around with this after the session. So this is what it looks like over here just want to see the front page of it. It says stuff like, you know, why do people flying their dreams. What's the flying limit thing you could do create a poem, right step by step instruction make pizza, you know, look at this though this is most important. Choose a conversational style, creative balance more precise, you can do it. You see in the bottom, you just type in the bottom your question. And you have to get more and more concrete about the questions. That's all this look like for copilot. But look at this. This is a very interesting analysis and I checked on, according to recent news article, Microsoft co pilot does offer free access so I checked to myself, GPT for and Dolly three. Those are the most advanced ones we have. However, it appears that the app has not impacted the population entirely, because people are still buying $20 one, even though you can get for free. It's a little bit bizarre but that's, you know, I want to just show you honestly that's the reality at the moment because I think people just not familiar enough with the options. So they're going with the tried and true tried and true, you know, over since what a couple months for the one and paying the money. So anyway, it's just an interesting phenomenon is not just not catching up with that pattern of behavior in terms of decision making yet. So I'm trying to share what I'm learning with you as best I can because there's distortions and misunderstandings and I don't know everything by long shot, but I learned a lot between courses and you know, delving to myself, etc. That we need to share with each other. Anyway, back to tools. So there's Dolly two, which is not as conversational in terms of that's the whole point of these things chat to each other's their conversational rather than being a computer language to be able to get these answers. So I use Dolly three because I work in tobacco prevention in Arkansas, as Argo as keeps helps keep kids away from tobacco and adults, minority kids for ages now, and I'm working with them to help them do empowerment evaluation work. And evaluate their own programs. So I wanted to show an image of how important and powerful this problem or how serious this problem is worldwide because this problem of smoking in particular. I'm not aware of this but tobacco kills more people than you know suicides, AIDS car crashes, you name it combined, but it's a worldwide global problem not just an American United States problem. So I want an image that would be powerful and in my evaluation reports, and I asked this, a hand holding a globe with a cigarette and it works as a nice visualization using free Dolly three. Two tools than Dolly. I don't want to go into too much detail but there's plenty out there. This one's called KREA and look at the difference. This one I can create draw a 60 year old professor with white hair and white beard. Interesting, writing his new book. And that's not exactly me that's for sure, but it's more realistic in the portrait of a human being in this case. And it's not too bad. One of the problem with a lot of visuals as you'll see, they're not good with the hands. This one's okay. Many of them show three fingers instead of four. So a lot of things that still have to be cleaned up and improved, but it gives you an idea of what these things can do. I'm just showing you another alternate one than Dolly three so you know there's many out there. Now, I want to emphasize that does take work, and that I don't get it right the first time second time and sometimes not the third time when I'm doing this so you have to give it iterative prompts and learns from your prompt after prompt after prompt. The first prompt I gave you for this one, because I'm also doing a doing valuations, using a permanent evaluation in India, and we're trying to eliminate tuberculosis USA ideas funding it and one of the things I try to do of course come up with visuals to go with our reports to show what our successes are failures you know our errors for refinement. So I tried this one. The first prompt was a picture of a hypodermic needle eliminating tuberculosis. Well, that's not what I had in mind see how the, the lungs are like in the hand and what doesn't mean much and it's awful with words see it's spelling tuberculosis, you put it in there it will misspell it every time. So I try it again. I reword it to a hypodermic needle eliminating lung disease in rural India because a lot of stuff is in rural parts of India looking for states right now. In any case, look at that that's not really what I had the rural project to see the kids, but that's not going to be too effective for my report. The needles in the air that lungs hanging out there is kind of bizarre looking. So I did it again. I did a hypodermic needle in a person's arm have to be very little to eliminate lung disease in rural and that work. So you have to play with it and play with it to get closer to what you want. I just want to give you a concrete example of real life thing in my evaluation work that I have to do to get it to work correctly but when it does wow it's pretty powerful and free. So prompts, not a big deal. You need clarity, clear instructions line breaks sometimes. You want to specify the desired tone and or like you can tell that five pages two pages one page if you're doing an essay about something or a blurb or something style you want it to be a marketing style academic style reference provide reference texts, minimize but already you can see it can be self correcting split complex tasks. If you ask it that you might as complex thing is going to mess up. If you can break into pieces, you're in better shape. Chain of thought help model come reason its way to correct the answer if you get pieces it can connect the dots as it were tasks improved by measures performance like we do we're valuators and record the changes. I record even which product is as you can see and I tried to share them with you so that I can see where I don't have a quite right I'm learning how to ask the question better and better and better by recording it and remembering what went with what. Let me give you some demonstrations here because I know we don't have a lot of time tonight but this will give you a pretty solid idea of what you'll get when you play around with this. This is using barred the way to tell which ones I'm using which or anyone's using when they do this is their little cute little pieces here you know you see this little kind of top or that's multi colored barred. So this is goals. So all it is like type in. I want a five bullet history of artificial intelligence. It gives it to me early dreamers. See, and then has the birth of the field the 50s and 60s see and have what I've talked about Dartmouth workshop in 56. I just entered the prompt down here. Very simple. So I'll do a bigger picture of the same things you can see a little bit better. Big early dreamers, birth of the field, etc. Get the idea Dartmouth workshop, very straightforward. Chat GBT which many of you probably already familiar with you already playing with. I just put in you can and you can see how it's 3.5 and 4. When I say when teaching a class about artificial intelligence, let me make a little bigger for you. When teaching a class about artificial intelligence evaluation, it's important to cover key concepts, which we did. Introduction to artificial intelligence, we have data quality, got a valuation metrics, we've got ethical considerations, I'm embedding throughout it. So I'm trying to put a check on myself to make sure I'm covering all right things when I'm sharing this with you. Demonstrations, this is quad. This is long text better and I asked this, you know, for five examples how to use artificial intelligence to conduct program evaluations. Got sentiment analysis, got text summarization, image recognition. The works, it pretty much tells you what we're looking for an ounce of large qualitative and quantitative data sets, etc. I mean to think about me some of the stuff you're going to be amazing second I'll show you more stuff that can happen in seconds rather than days as I show you a little bit more of this. This one I use co-pilot, which I highlight at the moment is the best one. Once again, I know I'll be 100% wrong in six months, maybe in five days, who knows. But the moment this is the smart one to use because you get to use chat GPT and dolly three for free. So I mean, I don't know, to me, I'm not, I don't know what I'm missing here. Why am I going to pay 20 bucks a month, but whatever. This one you click on it, you know, more creative answers and questions that are associated with it, more balanced and more precise. I go for more precise for what we do for the most part. And look at this. It tells me right here, use chat CPT 3.4 and I mean 3.5 and then for Microsoft co-pilot, barred, clawed. So I make sure I covered the ground and make sure I didn't make a mistake. And you can download it to Word, PDF, text, and you can build on each question and remembers what you asked and can get more refined if it didn't give you what you really wanted. That's just, I did a blow up so you can see it better since we're doing zoom and stuff. Once again, it's just, you know, different tools, chat boxes that we're using. And applications to evaluation. Look at this. Draw a logic model of tobacco prevention program. Research and planning, community outreach, education, policy. So this is, I do a lot, as you know, empowerment evaluation. This is my new book. I know if you can see it right now, I'll show it to you later if you can't see it. This one's empowerment evaluation social justice. When I'm showing folks how to do their own evaluation, this book is great, wonderful book, great. But I can also show them to you how to use AI to do a logic model themselves. I can look at it. I can have other folks look at it to try and get it to make sure it's still clean and it works for them. But look what they can do with this that quickly. That's what we did just with this one on our tobacco prevention programs, which I'm sharing with them. So they can play around with this with themselves and build their own capacity. So not depending on our solar plan. I know it scares some people, but it just means that we can operate at a higher level and not just do, you know, management information system that should already be in place already. And then the stages of evaluation, same thing, theories of change I asked it to do for a tobacco prevention program using co-pilot. And boom, it gave me the stages of change over here of trans theoretical model, a planned behavior theoretical model. So they can get a theoretical model and she over here, for instance, the California Department of Public Health Tobacco Control focuses on interventions. A second ago, healthy stores for me stuff and they give you a footnote so you go check it out. The example is how to use it. Anyway, I'm going to keep on going because I want to cover a lot more just another five to eight minutes and then open it up. Applications once again to evaluation and there's a million of them, but I'm trying to give you as many that can quickly. This is interview questions. Provide a list of interview questions in an empowerment evaluation. Hey, of course, they should know how to do this. So they read a little bit, maybe do some workshop, whatever, but they didn't need to check on themselves by just simply asking at any time. What are your goals? What do you think the program, how do you think the program is working? What are the strengths? What are the weaknesses? So very straightforward. It's like, wow, this thing is, I don't know if they're going to need me anymore. You know, it's like very clear, keep a safe and supportive environment, open any questions, listen actively. This would be respectful of different perspectives. It's beautiful. You get the idea. Check this out. This is really cool. Also, I've been really having fun with this. I'm just using chat to be tea for this one, right? You can do data analysis. Okay, you can put a data set in or borrow and I'll show you how to get them as well if you want to do interpretation of the data set cleaning. You know, if you have like duplicate sets of information, you don't want duplicates to get rid of that unit and analysis. And it gives you a bar chart like this. I'm looking at salary trends for folks that do AI stuff and all that sort of thing. Here's how you do it. Say if you don't have your own data set. Kaggle, C-K-A-G-G-L-U has free data sets all over the place that I use for pollution and population control and a lot of things. And I said, all you do is I type in this. You know, I clicked on the data set that's available and I put it here jobs in different data set, you know, kind of different jobs. I'm looking at for what you call it computer science and stuff. I said, I uploaded that you click it into here. And I said, interpret this data. And when you have all the aspects understood, say I'm ready for the next step. I said the data has been cleaned by removing duplicate roles. It tells you what it did, which ensures better accuracy, et cetera, et cetera, stuff we already know. But it's doing this in like seconds, right? And there are no remaining duplicates in the cleaning database. Are there? This is a trip. And look at this. It's the salary trends by job category, salary trends by experience level, salary distribution. You can see if you look carefully, what a shock, machine and learning AI. Look at that. That one has a beautiful salary range over here. Who knows how long it will last, of course, but right now that's the top one for salary range. And others are going to start to disappear. This would take me approximately two days to do normally. And most of us, I think some of you might be faster than me a day or day and a half. It takes me about two days. This took a lot of time. This took about almost, I don't know, 40 seconds. Holy cow. I mean, this is like a trip. This is a game changer. Anyway, let me move. I know we don't have much time, so I'll go quick. You can use DoorDash. Dropbox has something called Dash, which you can use just on your desktop instead of exposing all your stuff onto the internet when you're using AI. And you can say, I just wanted to look at your computer, my computer files, just my email, just my this. And it will only look at those things and do summarizations of your information as you want. I'm just trying to show you can use this on your desktop already. Never mind paying $100 to $200 to have it tailored for you. AI detection, don't waste your time on this. People are worried, oh, students are going to be cheating. How good the detection is. Most universities, most of the training I get also on this and the different faculty roles I play are that there's too many false positives, too many biases, etc. And the bottom line, if we're worried about this, and I'm happy to talk to you more in the future, or look at John Nash's work. We're teaching wrong. If we're only asking it to do what it can produce quickly, like do a summary of, you know, this novel you're supposed to be reading in the best. If we're not doing it right. We should be asking about their thinking about how they are coming up with what they're thinking about. And if you ask them to search, say on the net for some topic. Tell me where the biases are and make the, you know, tell the student to look for the biases and tell me what they are. In other words, we have to reshape our learning process and our teaching. If we're that worried about cheating and that sort of thing. It means we're doing the wrong thing, but I can go into that more detail. It's not 100%. There are issues of making sure that students actually make you write things in class, etc. To deal with it at the moment to make sure we get their actual information, but I don't want to go into much detail now. I'm just saying this is a wonderful tool to teach and to learn from with the right guardrails. Let me give you some resources and I'll stop in a second, but this is really important. And of course I can send a PDF if you're interested. But here's something that I really recommend for YouTube resources. If you want to know more about being in chat. It's a beautiful little guide to being chat AI with dolly three in education. I would look at the John Nash's podcast and stuff. Big think the role of AI in education. If you want to know more about what I spent a lot of my time in in medicine. This is considered the father or grandfather of AI is the person on the right large language models in medicine. And they understand and have empathy. We'll talk about that another time about how they're training position to have better or more empathy in terms of responding using AI tools. And then the one you saw me highlight earlier about how good it is and detecting things. That's can AI catch what doctors miss. Beautiful piece. Check out these kinds of references if you're interested in more. There's plenty out there. These things I've been using to learn as much as I can in a relatively short period of time. These are free training resources. IBM has one a generate AI. I've taken the one with the night center, how to use chat to be T and other generate AI tools. I'm taking another one that Claremont's offer me and my professor over there. Find the free courses. You don't have to pay for most of these they're free learn this a tremendous amount in a short period of time you can gain. Some of the references I use today. You don't have to use these use whatever you want. Ask me about the later whatever you want to do. But I looked at six strategies for better results. I'm going to have to do prompts better. How AI chat book bots like chat to be T or bar work, a visual explainer, mastering Google bar prompts very important. It goes online. You get the idea right. You're not AI is here. What if therapy bots become too good. Very interesting article. The ultimate guide to data democratization and toward conversational diagnostic AI. Enough. I gave you a lot in amazingly short period of time, but I'm excited. As you can see, I'm sure you're excited. This is amazing. It's a point in time. It's a turning point for evaluation and for, of course, all of the industries. If you want more information, you can email me. I have a Ted talk that you might find interesting about empowerment evaluation in this case and the evolution of it. And then a brief bio sketch and set up. Let me cancel out of this. So I can get to some questions if you've got some questions that I can answer as quickly as we can. Thanks for hanging in there. David, we have a while we don't we don't need to rush we we can also go over time we have a good 13 minutes. There are quite a few questions in the chat. I'm starting from starting from and what impact impact with quantum computing have on these technologies. Oh, it's phenomenal. It's what's amazing about this is it is able to take internet massive sources, all of Shakespeare's work. I mean, you name it. And the speed in which it can sort analyze and identify patterns is astounding. What's nice about the kind of computing power we have is that we didn't have that then the algorithm wouldn't mean much. You still have to have the power behind it that answers your question. It's right now we're having a lot of synchronicity of the right things coming together to make this possible. The timings and is incredible. Oh, yes, can you have a presentation. I see that yes, I'll make a PDF, and I'll send to you and you can post it wherever you can if that's okay so folks can have copies of this. Absolutely. Yeah, and it has hypertext links in it by the way. So you should be able to link on to the things if not just drag and you know do it put it in an AI chat not necessarily Google ones that identify some of these things. Oh, sorry. Melanie had a question about. Yeah. Sorry, Melanie's question is about summarizing large quantities of data which things like chat GDP cannot do because of the word limit what software would you recommend. And it's in a CVS file or a text file. You can then upload it it can't handle where it has too much garbage in it. You know things you can't see but a lot of you know things that in the spaces and stuff like that and the programming, but most of you can. It depends on which one you want to use but most of these you can upload like try it with co pilot for example. You know the example I show with travel. If you have a data set that's in that format, you can just upload the whole thing. And if you have it on your own private. If you some of you maybe way ahead of me and you're already doing your own personalized data sets I'm playing around with that now. It's called a huggy face where it helps you program these things to your own space your own domain. And that's the case where you are protecting it at the same time as well as being massive data sets. Look at cradle you'll see examples of it and play with that first, and then do your own as well I recommend because I had to do that and really understand how to use it correctly. Okay, cool. Thank you. How do you get AI to make editorials of we have already answered that that's okay and this chat GDP co pilot evaluate the quality of the references it uses as part of its process. Yeah, it does. As of today, I don't know what to tell you. Next week, who knows what's going to change, but I triangulate anyway. In other words, I will look at the references. And then I will also ask it to please check on the references and let me know if they're all accurate so I actually am not only checking myself but I'm asking it to check, because that's part of it's learning that helps all of us. And knowing that because if it knows we're asking that, then it goes, Oh, I'm learning from that I better do that automatically. And that example I showed you where co product gave me the references and the, what do you call it the footnotes. It didn't do that before, before you'd have to go check them out or just use some stuff out and be half garbage. And you didn't know what it was. I mean I had one way you can ask, you know, I asked about myself, of course, and it said something about teaching in UCLA. I've never taught at UCLA right so it's like what the heck was that a fake reference. I found out what it was. My, my uncle is also named David Fetterman and he, no, here he's held them and sorry her opponent, and he not only taught at MIT, he taught in UCLA. He also picked up his name and code, you know, merged it with mine, when I'm asking about it. That's only two months ago, and now I do the same thing, and it knows the difference. That's how fast these things are changing. So if you ask me any of this stuff, two or three months ago, the level of inaccuracy was like it's astoundingly bad it was like frightening. And it's still bad, but it's like way better in only a few months for things of that nature. I've been literally testing it to see what's what it can do. So I started to give that much detail but that's the nature of the change is so phenomenal but it's based on our feedback about well, that doesn't sound right or could you check on that for me that doesn't sound correct and it will check in addition to adding say I want references. The more you tweak it with your own prompts, I would say is the best thing to do now. Remember, this is a baby that we have to be teaching is what we're doing. It's a learning machine. It's, it's, and it's only going to develop if we give it feedback on that looks racist to me. I'm very blunt about it, you know, or I don't think there's a gender balance in that picture at all. Can you give me something that's more multicultural that and it's learning from that correction that that argument I'm having with it. It's really a conversation. That's great. There's plenty of questions how would how would you manage privacy David. That's the way that is two ways, I would say at the moment. Well, three ways. One, you can get an enterprise account for chat GBT and other things like that what that means is it's protected. You're painful. It's not free then. And you can even create your own data set that it makes quality, you know, data in there. And it's protected in that spot. It does not feedback into net. One of our big concerns was exactly that and it was a massive lawsuit back in. I think it was December. So I was, you know, ancient history. That's what that's like, a couple months ago. Wow, that there was a big lawsuit about proprietary stuff. Someone put some stuff on there from their, their firm. And then they were, oh my gosh, I see it on that people taking it well. At that point, there was no enterprise account that you have to protect it. The second way. So when you pay money, you have a protected it doesn't go anywhere. The second way is with these personalized models, please take a look at those if you're interested for a couple hundred dollars now. I mean, between one and three hundred dollars, you can create your own like little space as it were cloud for your account that's protected on your personal laptop phone, whatever is the second way to do it. And that's why I say how do you face and other tools like that that I didn't have time to go into today are tools that we can use our level to make these or you can pay someone to do it and then not expensive. So that's the second way. And there are more other ways I can go into detail, but those are the best ways at the moment to keep the proprietary information you have protected in a key space. That's what companies, you know, Google, Waymo, other companies are doing exactly that kind of thing to make sure that they have protected space, because they still want the power of AI but they don't want all their stuff out there in the world. You know, because otherwise if you don't have protected, you are sharing it with the world is learning from it but it's also bringing out there, and you have to worry about copyright issues, attribution, a lot of things. So not perfect. Oh, go ahead. Sorry. Yeah. Interesting reviews on the level of transparency is required when evaluation outsourcing. Let's see. Again consent from the client to use AI. Oh yeah, you should be asking. Sorry. Let me read that for everyone. You see the one on a little fast on the second. Yes, I'm interested in your views on the level of transparency as required when evaluations are outsourced. Yes, definitely want to be very open about no question. I understand that some New Zealand firms are adopting data procedure policy. Absolutely, which they have gained. You have to gain consent from their client to use. Yes. AI and the delivery of the services, which is agreed and specified in the work of order of the contract. I would say, you know, it depends. It's not that simple. Yes, I would get permission whenever you get permission but they're different levels. If it's a very tiny little group situation it's a nonprofit you're working with, and they just started to get off the ground. I would say you want verbal permission you may need some written permission if you're going to publish or put it out there yes, I would say you want to be very transparent. But why not? I mean, I don't I believe in honesty and straightforward. I was going to come and invite you, you know, and why not you want to be honest with people. So I would say you be as open as you can. I don't think you have to have, you know, a written contract for every single. I want to take this small data set on nutrition and see what AI says about it. You know, you got to just use your judgment as to how, how, if you're getting to private information, confidential, absolutely sensitive information. It depends on what level the stakes are, etc. Air on the side of transparency, I guess is a nutshell, but you know, none of us can afford all the time in the world and all the consent in the world for absolutely every single question. So you just have to use your judgment. So what's what's really appropriate. And it's just your normal training. That's I don't think there's anything extraordinary about it. But good question, because a lot of ethical issues associated with all this stuff. And I don't mean to, by the way, dismiss all the problems. There are lots of problems. I just don't think we can ignore them. It's here. The state's not going away. And the best thing we can do is to help improve it by being immersed in it. And so many people are just afraid to bring that, you know, baby with a bathwater. You have to we have to jump in there and see what doesn't work and what does work and why and then have our own recommendations. We have to be part of that process of correction. I believe it's like you wouldn't have to keep your responsibility with a student or your kids or anything like that. Right. This is a baby. It can be misused. Everything can be misused. Every technology. So our job is to do what we can to better understand it so we know what we can give for input for guardrails and things of that nature. And that's my bias. I just want to make sure you I don't dismiss the problems. I'm very much aware and immersed in how to correct these things, but I wouldn't know unless I immerse myself in the process, I guess, in a nutshell. Oh, yeah, we've got the confidentiality thing. There's a lot of things like the, like the enterprise thing, which I don't miss anybody. We have a question by Paul, why how can we use AI to our advantage. Can you see that one David as experience evaluators. And provide, please feel free to unmute yourself and ask David the question directly if you want to. Yeah, sure. Thank you for this opportunity. Thank you David presentation was awesome. But for me as an experienced evaluator, I just wanted to understand like, I can understand if we give this basic command to produce results, but how do I use it being an experienced evaluator to my advantage to then further optimize it and take it in the track where I'm thinking but also in a more polished way. Oh yeah. That I should be aware of when I do that so that I don't get misled. Thank you. Oh, yeah. Yeah, I mean, so you're just asking so different ways in which you can use it. So you're not misled by the results part. Yes. So like, instead of asking, like you were showing some examples, that's a good start. But like if I have already certain structures and ideas. How do I work collaboratively with AI to take it to the next level with also getting my inputs in ensuring that it also follows my lease. And then understanding what could be the potential risk by just going with AI. There's so many it depends on what you want to do, but I have students right now and colleagues using it to summarize masses mass mass of data sets or like what I just showed you with Gregor or see what the patterns are. That saves me two or three days instead of they do it in almost almost a full minute sometimes it takes to do the exact same thing I have to spend a day and a half to do. That's one example of data cleaning, which is a pain in the neck who likes to do that. And it does it really well interpreting it, and then actually doing the analysis for patterns and stuff like that. phenomenal. But my point in the bad example is not only is that useful for data analysis uses a nice sophisticated tool for all of us and saves us a lot of time. It allows us to ask larger questions, because we don't have to spend forever just doing that part in normal evaluation, we spend the entire time just doing that getting paid. That's enough. Today, it's like teaching in general, that would have been enough for an assignment right now, you can get that quick now I can ask much larger questions about different kinds of in this case, computer sciences, and how much money they're making and what the likelihood of the projection is of one being something my kid should go into or not go into because it's not going to exist anymore. So I had my own practical value for my students, but I wouldn't even have time to think about those questions. It's the same as qualitative data software, in my opinion, you know, like in vivo or, you know, all the other ones where in the old days, and many of you are too young to remember this but people used to use little cards, little index cards and they list them all out and then they have to go Oh, there's this one this one and sort it by hand, and you have a pattern go Oh my gosh, that took you, you know, about a year, but you know and now you do data sets and you can do that like in about you know a minute and a half two minutes. Think of that exponentially it's the same thing. Now instead of having to read every one of those articles. Okay. It does it summarizes it for you. And now you can ask the larger learning question because now you see the patterns that are in there that you would have taken forever to find. And the real question isn't the patterns. It's what are the implications for the patterns that you see in that data, or specific ethnic group that you think is being disadvantaged or not treated correctly, or there's an economic wealth gap between that group and that group that you haven't identified before. You wouldn't even think about asking that because it takes so much time just to do the data sort. That's what's so beautiful about this process. Do we have to put a check on it. Let's put a check on our own work to see if we got it right for data analysis and sorts of their biases. This is no different. It's just a larger scale and faster. Other examples aside from your literature search to do that. You can also say things like I'd like to see what would my students dissertation or my own dissertation if you're a student look like in that there's nothing wrong with that. You don't want to steal from it. You want to say it gives you a logical pattern to start with about what you report your evaluation report would look like for the my case tobacco prevention, or trying to eliminate tuberculosis of our project in India right now. What would be the summary of information based on all the stuff we've collected for all of our empowerment evaluations, for example. There's still a lot of work. I mean, to go through all the different ones of product and the exercise we've gone through, but if I put them together in this and see this file, or did a text file or whatever. I uploaded to that. Tell me what the patterns are or clean it first. How many of the patterns are and boom, I cannot ask much larger questions like, wow. One of them in the rural area that I'm working in India, for example, I see the same kind of issues of credibility or trust of putting this, you know, terrible thing into someone's body when they're worried about vaccinations, for example, as another area I work in. In the central city areas, but with low income, I can see the pattern that emerged that I don't have time to play with. Oh, sorry, going so long, but there's so many examples of what you can use on a routine basis that make your life simpler. And it does a lot of the busy work. So you have going to open your mind to other things that are much bigger patterns. But anyway, sorry to go on so much, but think about. Yeah, yeah, sure, sure. David, we are even though we have five minutes over the time limit. I'm happy to go on if you are David. Tim, you have asked a few practical questions. Do you want to unmute yourself and ask David directly. Is Tim around? Yes. Hi, thank you so much. Appreciate the really interesting, exciting new field. Yeah, just following on another discussion, I think with Marie, how I just checked, how can you actually upload documents into Copilot or what other platforms did you use? Because I can't find that functionality. No problem. Let me just share with you one more time. If I could, give me one second. I'll be right back there. And I'll show you. It's simpler than you might think. Watch this and I'm going to just go back in a second here. Like so, like so. Let me show what it looks like. It's none of it's that complicated. It just looks like it's when we're not used to doing it that much yet. But you see what I did here? I took Craigle and just play around with that for those still new to this, you know, using data sets of the size. There's a ton of them for free that may be relevant to your work or related. Maybe it's not a chance for playing around with it. So, on Craigle, it has all these free data sets. I want to know some stuff about computer programming stuff like that and computer scientists and what kind of money they're making, et cetera, et cetera. So I just clicked on the Craigle data set I wanted. Okay. And then I, it then put it down over here as a CSV file over here. Can you see on the left hand bottom? And then all I did is I made sure I looked good. I want to look at it and make sure it wasn't garbage itself. I really wanted their salaries and, you know, what, where they live, you know, and the actual title for it. I know what to make of it, the title and the categories over that. So I did check it, you know, with my own eyes, not just rely on artificial intelligence. And then I went to over here, chat to BT, and I just put upload over here. You can see it on or you can see this, you know, the regular, what do you call it, attachment things or what do you call it, paperclip thing, whatever. And I clicked on it and see how quick jobs and data CSV file here. And then I just typed in the box, which is this box. You see this, the paperclip I just clicked on that and attached my file right there, my CSV file that I want to clean the jobs file on it. And then I just interpret this data. And that's how simple this is, I don't want to make it sound complicated. It's sometimes I think it's so simple you can't see it, you know what I mean. It's not the paid version David is it the paid version or is it the unpaid version. Oh, I'm paid. There's all paid. Okay. There you are. I'm only done with free. Now having said that you can pay if you want to get chat to BT for $20. I use it for free through copilot. I don't have that function and I've tried uploading and it says sorry I can't do that. Could it be a regional thing like if I'm using an Australian server functionality here is different or why would it be for you. It shouldn't be are using which one are using their chat to be before or what are you using the free chat GPT 3.5 but also copilot. I don't either have any way to upload documents. You weren't able to do it. See choppy 3, 3.5, 3.5, the earlier versions won't let you attach and do anything that has to be for. So what you want to do is, if you had chat to be for, then it has the same thing you just saw right there. So I use it for copilot and I use chat to be for through through copilot and then I touch that because it's the same chat to be. Yeah, it wouldn't let me maybe is it if you're logged in through your enterprise is limited functionality. I don't know it's not yet. I tried and it said apologize and said you can't and. Oh, which history do it do it again. Do a computer screen snapshot of what you did do a description and just email me and if I if I don't know the answer I'll tell you I don't know the answer. But if I know the answer I can give you once I have the actual picture of that I can then be more specific. I'm guessing though. It sounds to me like somehow you're going through the path of GPT 3.5 instead of four. And it's not a matter of the money because you can pay the money and you should be able to do it for we can also do it definitely for free I use in what he called so take do it again, take a computer screen snapshot. Describe what you're doing again so I remember to get, I get like, you know, literally hundreds of emails a day. So let me know what you used and whether it's for with these copilot to get there, etc. Then I'll know probably where the problem is. Okay. And like wants to know if I'm interested in training AI to interpret the inputs output and their relationship to outcomes from case notes of social human services sector. Any references of this being done, particularly if there's research institutions where this is a focus. What are they doing the input output for for, or are you asking about the, there's two kinds of questions for that. Tell me which one you mean is the input output as it relates to the supervised learning where you already know what possible answers are or the unsupervised one where you're putting in saying the information about it you're asking the net to AI to come up with what the pattern is. Is it the first or the second. If you want to unmute yourself and answer David's question. Are you around. It's not around. If you're not here email me. If you get this recording. I'm happy to if I know the answer I'll tell you if I don't know the answer I'll let you know. But if I know which one it is, I can then be more specific. And that shell the answer to your question if you ever receive this and see this later is that the unsupervised one is the one where it's going to be most generative AI it's going to be the one that's really going to tell you something that's based on all of the information out there. The supervised one is very delimited it would depend on knowing the answer possible answer that would then be matched with that description. So if you do have a choice if that's the underlying question, do the generative AI. So what you're discovering and searching is going to be what you really want for a more powerful answer than one that's already programmed to have the input output but if that's responsive to your question. Can you hear me. Yeah, so my interest is in being able to potentially train. AI so that it can interpret case notes that government might see in service delivery. So that it can understand the inputs and outputs in service delivery and interpret outcomes. And so I imagine that would be supervised initially, and then maybe unsupervised in the future. When you could, you don't have to wait. It depends on what you're looking for an outcome of course or product yourself but from what you just said, I would use the unsupervised actually, but you would be limited to the data sets. You would say like I did you take the file you want it up there. And you'd say, yeah, show. Yeah, basically, but do it in like kind of the order I did it which is, you know, cleaning the data, interpreting it so you know what heck it really means from its perspective, and then do the analysis for pattern and trends, because otherwise you're going to get some junk. I'm just telling you all the mistakes I've made. So I myself, I forgot to do the obvious we're all taught to clean right you know you're minding stuff. I didn't. The first one so I got a junk and I could see it didn't make sense to me. I went oh geez I forgot a step. Just do what we all are taught to do normally clean it up first, get rid of that we don't junk and all the other stuff. And then ask it to interpret so she knows how to tells you what it means because you know I don't even know what it means sometimes these data sets. And then after the analysis for patterns. So if you do that or that's why I emphasize in the slides, then you're going to be fine and I don't I don't think you have to stick with the traditional supervised because then you have to know the answer to what this matches up to. And you don't know the I don't know the answers to that kind of thing for those that's beautiful case notes on it. See what the patterns are. I don't have time to go into detail but if you read the article on therapy. It's similar. What do you do when you can put all those notes in for example about their psychotherapist put in, and it can come up with a better analysis of what those case notes are, then you as a clinician. Whoa, am I out of a job. That's what the question is in it, but it does that with with even confidential permission of that nature but that of course, put it in enterprise or put it in a personalized data set so you're controlling where it could go. But you can all that I would recommend it from what you said though to use the unsupervised. I think you're going to find it much more useful because you're tightening the frame of what it could interpret if you only use the supervisors to me. Yeah, yep. I'll email you about potential research areas where this is being pursued. Thank you. My pleasure. My pleasure. That's fantastic. David, do you want to give a plug for your calls. That's happening in a few days. Yeah. Thank you. Thank you. I'm teaching for the American Evaluation Association on February 7 and 15, one of their e study courses on exactly this. I'm going to highlight some of, of course, the slide we discussed today. And in addition, for the second part of it, I'm going to having everyone who's in the session. Use one of these chat boxes. It could be co pilot it could be who knows you know whatever you want to chat to the tea, and they're going to apply it. And then they're going to come back and we're going to all critique it and give ideas as to what to do next, or how to use this for a different purpose. Do you want to use dolly three or do you want to use something else, not quite that like Korea, or do you want to use, you know, this for doing a marketing piece about your evaluation work and your firm or whatever, versus an analysis or logic model or something that nature. We're going to go into that kind of detail of real application. That we don't really have time to do today for that course. So yeah, thank you for reminding me so two things to remember for those interested in what I highlighted already for common stuff. Use the book first this is the latest book and power variation for social justice and beautiful piece. It got an award already. It's only it was done tonight in 2023. It never happens all of us we publish stuff who reads it right. This thing, they must have loved it because it has an emphasis on our work in India, trying to eliminate tuberculosis, and also on food justice in the United States. Anyway, so that's pretty cool, but this will have your folks that you're working with doing collaborative participatory in the power evaluation, and then introduce them to how to use artificial intelligence with this to learn to do this themselves as much as possible, build their capacity. Thanks for that. Almost forgot those two things are important. David we can we can go on forever. On behalf of all of us and the AES I would like to give you a very I'm actually going to clap I found this one of the, like the most illuminating seminar so it was absolutely brilliant thank you so much for your time. I'm sure you'll get lots of follow up emails from people and thank you everyone for for joining us for this almost an hour and a half, not quite. Have I if I missed you out. We do have. I don't know how long David is willing to wait for just put your hand up, and you can say some final words if you if you really want to do so. And put something in the chat if you want very briefly. Wonderful having all the thanks for hanging in there. I know it's a little bit long, but I want to share what I'm learning. We're all learnings together. And it's evolving so fast. I thought it's important as a community we just get the best team along the weekend. We're going to be in charge of raising this little baby. So, sorry, David, I've got one quick question. Sorry, Eva. Hi David. What's the LinkedIn group that you said that you did you say that at the beginning of the session. There's a LinkedIn group. Oh yeah, thank you. It's called, it's just called IA artificial intelligence AI and evaluation. And they'll look like this really quickly I'll show you this. Share this with you real quick. I'll show you the actual like this. And let me just show you what it looks like right over here. I think this is the notifications. Let me see if that's the one right here. Find one for you. And this is the one of things on this. So many files for you here. This is the latest one here that we posted on. This is AI and evaluation. John Beck is helping to coordinate an AI and evaluation over here and you go like this if you click on this. And then you'll see we have a ton of things going on. We have he posted a piece on chat TV T evaluation and research that they're charging for for that session. And now how to do an analysis like I was talking about today. We've got other pieces that people have highlighted that they're working on. We've got the oh this is the session I'll be giving for the American Evaluation Association cool picture. And so and this has articles about everyone's learning together we're just all posting on this site and just join on in is no no no nothing to no payment or nothing we're just learning together. It came out of our session that we had a we're so excited and so thrilled about sharing. We thought let's do it this way and not wait till next year it's going to be too late. So we're constantly just adding to it so please everyone feel free to join in your playing around with it. One last question. David is asking about participatory methods of using AI. Do you want to do you want to unmute yourself. Yes, thank you. I was just wondering you know it's it's amazing to see AI doing all that work but obviously we're working in a participatory collaborative training ownership approach and I just wonder how to navigate those two. I would say this so much with, for example with our work, we've been using the net a lot already, and this is just the next level of using it to help people build their capacity. So they can use it to learn how to do logic models themselves. They can do some of their own data analysis. If you show them what to do, and then show them the tools of how to use this and how we use it ourselves. They can see how to use it with their data sets. There's a ton of things that it's just perfect for any kind of collaborative participatory and prominent posts because you're constantly building their capacity to do this better than us for some parts because they know the area better. And we can be a check and balance and help them make sure it's rigorous on target check for weaknesses associated with the data sources. We are very valuable at a higher level, but they can do a lot of this themselves. And once we share that these tools with them, which is why I'm constantly trying to learn so I can help others use this to do their own assessments. And even if you're doing a traditional evaluation, look how much this saves you into the data analysis data collection. This is the sort you can do easily, I mean, without cost, which is amazing, but the speed is the amazing part of it. So even if you are doing traditional forms of evaluation, never mind collaborative participant empowerment. All of these are powerful at helping us move faster with what we're doing. Quite frankly, like the physicians are learning probably with more accuracy than we have for a lot of our work. So we still needed absolutely to put check on all of this sort of thing or expert judgment, etc. But the head, the, the, the, as you're hinting, the father or grandfather of artificial intelligence said this, which is not going to go well and scare a lot of people. He said, the human brain is amazing machine phenomenon. But there's no reason to think it's the pinnacle of intelligence. In other words, computers might do some things better than we can. At this point, we're still in it. I know no one wants to hear that that might be out of the job. At this point, physicians are already seeing that this is going to be invaluable to work underneath them, right. That doesn't mean someday it might be that the artificial intelligence is what you check first for polyps for x-ray issues, and then have the physician judgment to check on that, but then to work at a higher level. And then it's all able to operate. So, instead of being afraid of it, it's more how can we use this so that we can maximize our potential rather than being afraid of it taking over. It's just a different frame on that. So I just want to add that that's coming from Godfather or grandfather of artificial intelligence. I've been trying to learn a lot from people who created this, what they're thinking is above and beyond our specific techniques that we use for evaluations very interesting. So I hope that's a helpful way to thoughtful kind of extra extra comment or thought about the larger picture of what this is all about for all of us in our lives. David, there's demand for for your training in the quantitative area. There's more demand for you to give workshops down here as well. And so we will we will definitely be in touch. Thank you so much, David, for your time and thank you everyone for coming. It's been a wonderful wonderful educational informative session. Thank you. Thank you so much. We thank everyone else. Thank you. Thank you. Thanks everyone.