 Fiora Kautou-Katoa, Naumai Harimai, welcome everyone for getting to grips with artificial intelligence in evaluation presented by Dr. Paul Digen. This seminar is recorded and it will be uploaded to the AES YouTube channel in four weeks. I am Marina Sanker, the co-convener of AES Aotearoa and my colleague, Rula, will monitor the chat section along with Paul, who can also see the chat section, which I cannot at the moment. So we will also have 15 minutes at the end for Q&A, but in the meantime, do feel free to put your thoughts and questions down in the chat and it could be addressed as the presentation is going. Before we begin, I'd like to acknowledge the traditional custodians of the lands in which we all come from. I am speaking from Poneke, Wellington, Aotearoa and I acknowledge the leaders past, present and emerging. In tradition with the customs of Aotearoa, I'd like to open this kōrero with a karakia. He waka herenga, he fide fide fakaro, he fide fide kōrero, kaute maramatanga, tiihe mariora. As evaluators, researchers and policy experts, we are now all in a position to make a decision of if we use AI in our work and if so, how we utilize it. It is therefore important that we have a clear understanding of what is AI, what it can do, its benefits and its limitations. So today, we are all very fortunate to have an expert with us on the subject matter. Paul is a highly experienced evaluator and trainer who also has a background in psychology and technological social impact assessment. He has been a senior full-practice scholar at the Urban Institute in Washington DC and has worked with numerous organizations from community-based groups through to the International Monetary Fund. And most importantly, he has just finished writing a book on AI, which will be out very soon. So Paul, we are all so excited to hear from you. Over to you. Thank you very much. Kia ora everybody. Morena. So I'm Paul Duggan from, talking to you from Wellington, New Zealand. So it's the delight to be, I've had a lot to do with evaluation over the years and then I've been doing a lot of other things, a lot of strategy and then recently doing a lot of things about AI, but it's great to sort of meld those things together into this discussion here. So there's sort of a lot of the favourite, my favourite topics are coming together here on this discussion today. So welcome everybody to this webinar. So I'm going to basically, I'll just get the, if I'm not as Muslim, I'm not exactly sure who's moving the slides, but whichever one of you is moving the slides, could you, oh thanks very much for it. Could you just move to the next slide please? Thank you. Brilliant. So what I'm planning to cover in this webinar is, first of all, what can AI do for evaluators? So I just, now it's very hard to demonstrate this because we've got quite a lot of people here and everybody will be kind of, have different experiences of how they've used AI so far. So some people will have refused it exhaustively and some people won't. So it's a little bit hard to know exactly how to do this, but I'll just run through a demonstration fairly quickly of some of the things that AI can do. Then I want to talk about the implications of AI for evaluation practice and I think there's a lot of them, apart from just using it as a tool. And then the social implications of AI for evaluators, which as Marie said, that's the book I'm writing at the moment. I'll just, just finish writing, which, and I'll be talking about that. Fantastic. The next slide please. Brilliant. So I'll put up a resource page for this webinar. So if you go to bulldygon.consulting.p50 and I put it in the, in the chat, you will find some resources, including the verbatim session I had with chatGPT preparing this material I'll be talking through. So as I talk through it, I won't be able to, you know, discuss every detail of the, of what the responses I got from chatGPT, but you can go back at your leisure and look on the site. So this, this page has just got, it's got it. First of all, it's got a, it's got the chat session that I had with chatGPT, I'll talk to in a moment. It's got some other resources there. One I just want to highlight is if you could move to the next day, the next slide please, I've got a page on my website, which is called at bulldygon.consultingaiexamples and this shows you where AI is headed. It's really quite important for you to understand where AI is headed. And so this set of examples on that website page, don't look at it now, but go back and look at it. It's got a lot of examples of where AI is taking us because AI is like a many, it looks like a hydra. It's got many heads, many different types and styles of AI. So it's, I'll be talking about what a chatbot can do for us, but there's a whole lot of other types of AI, which are really, will be relevant to evaluation in various ways. Thank you very much. Could you go to the next slide? Oh, thanks for someone put that up on the chat. Thank you very much. Okay. What can AI do for evaluation? Well, the idea, the answer is it can do lots of stuff for evaluators, a huge amount of stuff for evaluators. And I'm just going to move very quickly through some examples of what it can do. So if we could move to the next slide, please. So what I did, I started the session with chat GPT. Now it's just a chat GPT for, if you are going to use chatbots seriously and there's a lot of different chatbots and they're already, they're appearing and being, for instance, they're appearing in Google. And there's a lot of chatbots out there, but I've just used chat GPT for, but if you are going to do it, it's worth while getting the paid subscription because that lets you use version four of chat GPT, which is I think something like 30% better than version three, which is the free version. Three, which is the free version. So I'd suggest if you really want to use chat GPT or chatbot for doing evaluation, that you get the latest, the latest version on it would be just one for a starter. So now, so I asked first, I think I asked it, so I got into a session with chat GPT. As I said, these sessions, if you had the paid version, they are captured in the chat and then you can share them. And so you see on that resource page, you put up, you'd be able to click and you'd be able to look through in more in detail that the answers are okay to my questions. So they're up there, which you can look at later on if you wish. First point about talking to a chatbot, you can make it be something. So acting as an expert evaluator. So you sort of set the scene for the chatbot, so you're saying rather than just asking the question, acting as an expert evaluator. Now, what I said is right instead of outcomes for a webinar on AI, a program of evaluation, which is actually just for fun, it's this webinar. So I've asked it to write a set of outcomes, which is something that we use a lot. We use it if we're thinking about implementation evaluation, which is formative evaluation, thinking that we want to help the program be well implemented. And of course, we need it for outcome evaluation. We need to have some outcomes. Next slide, please, Mary. So what it did here was it identified a set of four outcomes. So understanding AI practical application challenges in future direction. So it gave a set of outcomes and then some outcomes under that. So you can just see it can produce outcomes for a program. And this could be any program. You could ask it, it can produce a set of outcomes. I'll talk later on about how accurate all of this is. But it's having to skim through it. It's not too bad. And you can have a look later on yourselves, you know, sort of like about 80 percent, you can imagine making a little bit better. But it's sort of as good as many, many kind of evaluation reports that I would read the material in general that it produces. Next slide, please. So then I said, you know, as part of formative and process evaluation, we would want to make sure that I. That's right. So someone's just commented up on this chat that these lot more like actions rather than outcomes. That's right. So you're going to you don't ever want to talk later about this. You don't want to take what Chatbot says absolutely at face value. You need to critically engage with it. But often it'll give you the opportunity to engage with it. Exactly about that person saying, are they actions or are they outcomes? So then what you could say to chat GBT, write them more as outcomes and see where it came up with. So very much a matter of engaging in a dialogue with them. The next one is evidence based initiative formative and process evaluation briefly summarise the evidence based principles for running such a webinar. So that's we need to make sure the programs are based on evidence based principles, both in the formative aspect and in a process aspect. So next slide, please, Maureen. Yeah. So what it did then was it provided some evidence based information about how to run a webinar. I won't go into it. It's up on that should be up on the link. But it produces something as I say, if you read, I've just skimmed through these, if you read it, you know, it's a, it's definitely a starting point. It's not necessarily a finishing point, but it's a starting point for sparking your thinking about this next slide. Please, Maureen. So then what is it was right in methodology section with the evaluation report explaining the qualitative, the quantitative analysis, plus the use of grounded theory in the analysis of qualitative responses. And so then the next slide, please. So then it came up with the methodology section. And again, we should not take these things at face value and just sort of fire them out to a client. But they do provide a really good starting point or sort of like a first draft that you'd have to, I would suggest you'd never, never send a word out from, from AI, which you haven't checked yourself and thought about very carefully, but it sort of takes you quite a way down the track run, starting with a blank piece of paper, you actually, or, or often we start with an evaluation report, we've written about a different subject. This actually just really starts your thinking and gets you going in terms of that. Next one, please. Okay. So then often this is just showing, showing how many people say, you know, AI can't be, can't be critical or can't be analytic. Here, I've said justify the evaluation in terms of the OECD DAC evaluation criteria. So you can actually get it. So in this case, it goes away. It finds what those are and or it's got them already within its large language model. And it actually compares the methodology we've come up with, with those, that criteria. Next one, please. So there they are again, we don't have time in this webinar to read them all out, but they should be up in that, that transcript or just somebody asked, you know, just asked you if you did it again yourself. But here it's gone through and it's looked at each of those evaluation criteria and it's actually given a rationale for why our methodology actually meets those criteria. So if you think about that in terms of justifying what we're doing as evaluators, you can see how this again gives us a real good start in terms of that sort of more analytical thinking, not just summarizing it section. Next one, please, Marie. Okay. So now we move on more on to methodology. So so far we talked about methodology, we talked about outcomes and now we're moving on to the actual nitty gritty of the methodology. So I've said write a seven item post webinar questionnaire for webinar participants. And so next slide, please. So there it is, it's produced a seven item questionnaire, which again, I haven't had a lot of time to go through, but we can skim through that. Right, there's great. So someone's just posted, is it Katie? Someone's posted there about it can help you go become writer's block, brilliant point. Yeah, exactly. It's a starting point. And then you go through and you check. And sometimes, you know, I mean, I must say, I feel it's kind of 70 to 80 percent, you know, good enough about a lot of stuff. Something times it makes it something that's really just a monstrous kind of crazy thing. Or sometimes it's subtle like that first person on the chat said, are they outcomes or are they processes or activities? So so sometimes there's subtle things we have to do in terms of working with, but it's very much a matter of working with the chat, but rather than thinking it's just going to produce all the answers. But it does help with things like writer's block. Next one, please, Marie. Yeah. OK, so I just want to sort of show you the this is really just to illustrate where you can go with this and obviously you should play with it yourself. But, you know, you've so we've got a survey and then we go, well, detail how one could set the survey up and survey monkey, because, you know, we as evaluators, we have a circuit come up the survey fine, but we've actually got to get it out to people. So I may want to put in survey monkey. So, boom, next slide, please. It'll give you a detailed description of the steps you have to take to actually put that question there into a survey, you know, going blow by blow through the different aspects of that questionnaire, how you can put in the survey monkey. Now, the point I want to actually illustrate about this is that this is not restricted to it telling you how to do survey monkey. It's actually, you know, it could actually tell you how to put this into Google forms or all sorts of other places. But not only that, what if you go if you look later on in my page about where AI is going, that resource page I suggested. In actual fact, the AI is now being able to actually use software or interact with websites. So we're not too far from the point where you say generate a survey monkey and things are moving so fast. This may already be the case with a plug into chat GPD, where you say actually generate the form on survey monkey. That's not impossible, not even difficult at the moment for that next step to happen. So if you see where this is kind of heading in terms of not only the chat bot, you know, telling us how to do things, but actually then starting to do things and I'll touch on that later on in the final section. Next one, please Marie. OK, now this is this is sort of just taking a slightly different angle, but this I just really want to illustrate what this what it can do if you get kind of creative. So I'm trying to demonstrate to this to you guys and then I think I need a I'd like to have a sample set of responses because I wanted to analyze this later on to show you how I can do analysis as part of evaluation. So I just said to chat GPD, create a mocked up set of data for 20 respondents from the survey in a common eliminated format. So it's now going, it goes away. I won't demonstrate it here. It just goes away and it actually creates a fictitious set of responses, including qualitative and quantitative responses to that survey that it just developed early on. So this is entirely fictitious, of course, but I want that from my point of view, I just wanted this set of responses from an evaluators point of view. You can see one of my sort of feelings about evaluation is and particularly questionnaires is we should never design a questionnaire unless we've thought about how we're going to report it. OK, whenever I design a questionnaire for evaluation, I don't just design the questionnaire. I go, what do I want to write in the evaluation report? Obviously not having the answers, but I want to know that my questions will provide me with the kind of data and information I need for my final report. So in terms of this, you see you've got a set of questions. So then you can say, chat GBT, think up a set of fictitious responses for this and then you can make sure that when they were analyzed, they would actually they would answer important questions. We're not talking about the content of those answers. We're talking about making sure we've asked the right questions. So you see how you can use this as part of your creative process of developing the evaluation. Yeah, Mark, as Mark said, very handy, very handy, yeah. So it's and I'll talk later on about things very kind of fluid about the way you use your interactive chat GBT. Next one, please. So I went away and did that and there's a little it's very interesting to watch that it actually goes away. It uses a thing called Python. It actually goes away and uses a piece of software. So people kind of think chat bots can just produce language in order to create that set of data. It went away and used a programming language called Python. I didn't tell it. It just it just went away and it knew that was the best way of doing it. Went away, created that within Python and will show you the code when it does it and came back with that that which you could save as a CVS CSV file, which you could then pop in for Excel if you so wish or do anything you like. But really what I'm trying to say is this thing is this will get on to the end. But you know, the way AI is going is not just a very static thing. It's actually a very fluid thing that's linking to a lot of stuff. Fantastic. So I see it set up using the mock up set of results. Could you do a grounded theory analysis of the qualitative responses? So let's go. Go to the next slide. On the next slide, please. Thank you. So there it's done a grounded. So I actually asked, you know, it knows what grounded theory is and describe the methodology and ground theory. And then it's done. It's done a grounded theory analysis on the fictitious responses that it created. OK, so this is sort of boxes within boxes. No, it's pretty pretty amazing. But that's that's what it's done. And again, didn't look bad for a quick eyeballing it. You know, I didn't look it for long and you can go back and have a wee look at the chat session from that website. But you can see you can see some of the potential here of it actually doing qualitative analysis. I'll touch on that later on. Next one, please. OK, someone said they join late. Please tell me if this is the free version or the paid version. Thanks very much. This is the paid version. And I just said at the very beginning, if you're going to use it seriously, just like any professional, you really have to have a tool that is going to work. It's not not worthwhile using chat chat GPT three. You need to go to chat GPT four, keep up the latest version and you really need a paid version, which also enables you to depends whether you trust it or not. But but basically the free version of chat GPT will train on what you put into it. Whereas the paid version, you can tell it not to train on the data you put into it. So while you don't want to put any confidential, you know, absolutely confidential data into chat GPT, at least the paid version is not going to train on that. So it will they say it and I trust them it's not going to train on that. So for any professional using wanted to use chat GPT, they definitely should get the paid version. It's about $30 a month. It's also faster. It's just like any if you don't want to use a professional tool, you really need to even have to pay for it. But the main point is it's more intelligent than three. I think it's 30 percent more intelligent than version three. Fantastic. So here we're going quantitative, analyze them by grass for this far showing file showing the results. So there's just for that new person is right. This is a fictitious file of the results of an evaluation questionnaire that we've got chat GPT for PM Murray, the next one, please. Thank you. Cool. So now this is just this is just what it created. Right. So I didn't tell it to use any graphing program. I just said draw some graphs and give us some a summary of that data. And so it's drawn graphs. Now, again, you wouldn't want to just throw these in a report and whip it off to a client. You want to examine them to make sure that they make sense. And it's done the right thing. But this is a very powerful tool for playing around with the way you may want to visually represent the data. If you know what I mean, you can you can see what there's three. There's, you know, do I want to do I want to use that graph? Do I want to use that graph? In the past, you'd have to manually go and create those graphs, do the analysis and then go through that takes a lot of time. It kind of goes, hey, well, I think I probably want some graphs like this, let's, let's fire them up. And then you say, I want to use this one, I want to use this. I've just got a little thing caution needed using this with the ethical issues. In addition to data sovereignty in regard to the use of this, absolutely, absolutely a really good point. Thank you very much. Kelly, I think it is just we have. Yeah, so what I'm what I'm doing is kind of an enthusiastic way of using this, but you know, I use I got it to make up the data we're using here. So I would be very reticent to just dump in a whole lot of data into this with absolutely not enough people's names and things like that. If it was anonymous data, maybe, obviously there's a question of indigenous data sovereignty, which we need to think about very carefully. So thank you very much for that question. Kelly, I think was fantastic. So I'm really sort of demonstrating what it can do, but it's very much the potential we have to risk manage all of the stuff because otherwise we could end up in all sorts of muddle. So there you got it anyway. So as I say, I didn't tell it. I just enjoy some grass and do some analysis. So look at the data. Thank you very much. Next one, please. Cool. So this this leads on from what I was just saying, Kelly's point that I've got six rules for using chatbots and I'll just run through them. So this is also up on that resource site, the ad by website P 50. Now, first of all, don't ask a controversial question. So there's all this talk about bias and things and it's got biases in it because it is trained on the internet. And we know, you know, we know how crazy the internet is. Don't ask a controversial questions because you'll get nonsense and either nonsense or it's got to also come up. So there's no point discussing controversial things with it because you're just going to get you just going to get, you know, either they won't respond or you'll get you'll get biases. Secondly, I know this sounds paradoxical, but don't ask any question. You don't really know the answer to or you're able to figure out whether this answer is credible because it does this thing, which is called hallucinating where it makes stuff up. It's like a very, very keen intern on someone who will just make stuff up for you because they want to please. It's very, it's like very, very keen to please. That's like a big labrador for wavy tail and it will do anything, anything that you like. So you want to be in a position where you can evaluate whether what it's telling you is accurate or not. You really need to do that before you can trust it. Well, this is with the current set of of of AI that we're getting. This is likely to improve over time, but where it's like, get better. Now, rule three, talk to it as a conversation, not a one off. It's not just a matter of asking. This is a bit of a chat box. Don't just ask it one question. Think of it. See, as I went through that, I had a creative discussion with the chat box as we went, we went through. That's number three. And then think creatively to discover what's called capability overhang. Capability overhang is a scary thing. The people who build these systems don't know what they can do. And so it's really up to all of us to push the limits to see what it actually can do. All of us have their own ways we may want to use their own backgrounds and our own thinking about risk management has already had in the chat is to really sort of work these things to see what they're capable of and see what they're not capable of and see where they're making stuff up. But think creatively about the way you interact with them and just fire all sorts of questions at them as long as we are risk managing around confidential information, obviously. But have a real go at it. It's really, really is the main message from that. Now, next one is be careful about putting out a lot of AI techs. There is the 10 patient, you know, just what we've done. We could actually sort of just fire that all on to report, put out logo on the top of it and fire it off to a client. Now, that's you have to be very careful about that at the moment because first of all, I think yeah, I because there is the potential if you put out a lot of AI stuff that particularly published it on the web that at some stage things like Google might say, oh, this is all AI text. And you may be downgraded in a sense for that because it may be the AI text comes to be regarded because of the way people abuse it as of not particularly credible and not particularly useful. So so for instance in my book, I'll talk about my book briefly later on. But I've got at the bottom human written book because I think there was this value and it is a human written book. It's not an AI generated book. There's value. So we've just got to be read. I just really need to risk manage that. Now, it's so the question is if can you can you put a whole lot of AI stuff in a report to a client? I think if you discuss it with the client and you and you acknowledge that, then that may be fine. They may be perfectly fine. Say, look, I use the AI to help write the methodology and then I go, that's cool because it's sufficient. But you really just want to be clear that if you just pump out AI techs for the little proviso to that is what's about to happen with the thing called co-pilot, Microsoft co-pilot also in Google Docs. Really, everything's about to become permeated with AI techs. Basically, every email could well be written by AI as our emails as AI elaborates. They're just starting to see starting to offer more and more detailed suggestions about about emails and things. So so kind of got to watch that space. But just just sort of be a little bit cautious at the moment about pumping out. Obviously, pumping out just AI generator reports. It's not quite as easy as that. And then lastly, just try out AI. So what I've talked about is just simply chat box. So for just one one aspect of AI, there's a whole lot of different types of AI out there. Image generators, there'll be visualization generators as music generators. But there's also AI being woven at all sorts of different systems. So so the main thing is just and I'll talk later that very end about not, you know, overcoming AI anxiety, but really really just just try these things out because you need to know about them and we all need to know about them. And only if you try out and we will be able to be able to get more informed in regard. I'm just going to stop for a moment and and have a look at these questions. Cool. So we've got is the data still yet. So basically this model was the chat TBT was trained up to 2021. So it's sitting there idling those stuff to 2021. But what actually one stage chat TBT was able to do, but they just turned it off at the moment. But as something like being or also bad within Google, it it's a chat box with with that's when it learned its language. It's a little bit like when did it go to university? You know, someone went to university about about this time. But what they're doing now is they're wiring up the that language model, they're wiring it up to being able to look up the Internet. So some of them can update. Now, as I say, chat TBT used to be able to do that, but I've turned it off just at the moment because they had some problems. I think of the summarizing websites or some sort of quasi legal problem. So the point is, yes, it's only up to 2021. But if you get that's that's chat TBT. But if you get one that can actually look at the Internet of which part and thing the Microsoft part one and the Google thing one. Sorry. Bing and Bard. These names are very similar. Then what then it does actually look up the Internet so we can provide up to that data that's in there. Have you asked it to create some creative graphic displays by aggressive but boring? I haven't just I haven't asked it to do that. That would be really interesting. So maybe someone if anyone's got chat TBT, they could try. They could try it out until it's what comes up. But you're really good point. It can. Chat TBT has these add-ons where it can access other programs so that you can get it to draw a source of diagrams and it goes away. It tells another program what to do and goes away in case of that program. So there's definitely something that's coming with it. Exactly how it's implemented at the moment, not sure. Some point is the free access to the machine. Clearly Sergum Scribe. But like you say some good links that someone put up there. Thank you very much. And someone said my 17 year old asked Snatchek AI to summarize the first Harry Potter book in emojis. Totally useless, but very funny. Thanks very much, yeah. So that's really stretching your head around what these what it can do. And that's exactly the type of thing that people should be doing just to see, just try it out, see what happens. Maybe we could write a whole evaluation report in emojis. So it's probably kind of fun. OK, next slide please Marie. Cool. So OK, so that's just a very quick run through of how you could use AI to augment our evaluation practice. There's a lot more to it. You can see the transcript of that up on there on my website. My website slash P 50 where I've just put some resources for this talk. And just have a wee look through that and see what you think and see what you think of what it's come up with. And also if you can get hold of a paid version of it, really have a go at it and try out all sorts of things. But again, with the provider as other people in the web now said, don't don't put anything confidential in there until you can be assured as a secure environment. Just on that because that's so important. A lot of people are now working on having secure environments. Obviously, New Zealand government would be or any government's interested in doing that. So it will I it won't always be the case where the environment's not secure because people will get locked down environments where they can put secure data. But until that comes along, we should be very wary. But we don't need to think the chat box permanently will be insecure in terms of of the data we put into them. OK, implications for AI. Now, there's a lot of implications of AI for evaluation practice. And I think we need to think through these and they will emerge over time. So let's just jump into a couple of them. I've talked about implementation process and outcome of evaluation here. Marie, next slide, please. OK, first is in regard to implementation, evaluation and implementation evaluation. We're always thinking about whether programs this is just sort of a snapshot of the couple of ways I think that AI will affect evaluation. That's not exhaustive. We're always wanting people to be evidence based. So here's just this is just an article about data and decision making, how AI and data calls can help influence evidence based policy change. This is actually sort of in the social and environmental area, something called the Scottish AI Alliance. But the main takeaway point here is that it is potentially incredibly good at collecting evidence and it's very likely we'll get chat box trained on bodies of evidence. People are trying to do this at the moment because they're a little bit wild. They kind of make up evidence, but I don't think it's going to be the case going forever. But the first point really is if we want as part of implementation evaluation to make sure that people using evidence based policy, evidence based programs and initiatives, I think AI is going to be a big game changer there because at the moment we kind of just rely on people being able to pick up on evidence. A lot of our job in a sense is trying to work at did people use evidence based program logics? There is a change. And so I think AI is going to make a big difference in regard to that space that increasingly there will be the tools to make sure that evidence based policy has been included in initiatives. Next one, please. OK, so the next one, that was implementation evaluation. So next thing in terms of process evaluation. Now, the process evaluation is obviously, you know, describing the process. I see that as I was the way I define it is process evaluation is looking at the course and context of a program. But particularly about the actual details of the program itself, AI offers the opportunity for real time monitoring of all activity within an initiative. So if you just think about that, so at the moment, we retrospectively often go back and or sometimes we have been involved in process evaluation, which is, you know, we're going at the same time, but we end up glaring through documents, we end up interviewing people about what's happening, etc. AI holds the possibility of a real time monitoring of what's happening in an initiative. And this is really, this is really kind of a quantum change, a real step change here we should be thinking in terms of the value of practice. So all of a sudden, because in a sense, we could evaluate like a spotlight, we kind of apply a spotlight. But in the sense, this is saying the lights may be turned on all the time with AI monitoring what's happening in the program. Now, I just want to show you a piece of software or platform that that talks about doing this. So next slide, please. So there's various initiatives about this. I know there's one that there's even one or one interesting called Hoist, which is working on this this type of thing. But this is this platform called a CODA, which is the CODA, CODA, which is an AI based platform. Now, from their blur, they say that's the world's first ops intelligence platform. And they talk about it being a new standard for modern operations visibility. What this thing does is continuously just monitors the entirety of what's happening. Obviously, just what the organization wants to give it. But you can say, look at all of our emails in real time, look at all of our documentation, it's governed about, look at all of our legislation and constantly monitor this. And what that's what it does. And then it actually what it does is you see what the customer says, continuous insights into alignment and guidance on actions. And that whole back content of insights into alignment and guidance on actions, really summarizes process and formative evaluation, doesn't it? It's really what we're trying to provide insights into alignment between activities and outcomes. And we're trying to provide recommendations about action. In a sense, that almost describes the process evaluated in the implementation formative evaluators role. So here we've got AI in a sense, offering to do something very similar to what we do. So this is a very, I think it's a very, very interesting development. And I think we'll see more and more of this. So as a bottom, isn't this basically process evaluation that it's doing here? And in that case, what it does, this thing's monitoring. And what it said, one presentation, I saw about it, basically said, we'll monitor all of everything's happening in the company. And then we will report on, like they called discrepancies or whatever, you know, management by exception. And they, in that case, the example they gave, they said, we could tell you that for these customers, you are spending a lot of time on these customers, but you're not getting much profit from these particular customers. So that I know that's in the commercial context, but for evaluators, this is actually the kind of insight we provide from process or implementation evaluation. And here AI is spitting this out automatically. So, so I think that's the space to watch real time monitoring, in effect, process evaluation of programs happening. Next slide, please. Oh, sorry, just before we do that. Okay, so isn't this too dangerous in the big brother and real life? Thank you very much. Absolutely. Let's just go back to the previous slide, please Marie. Just give me one moment. Yes, so, well, there's several points there. Yes, then, so there's the whole sort of big brother thing happening again. This thing, a coder would be paid for by companies, you know, that they would say what they wanted it to look at. But there's a whole ethical thing here around, around surveillance. Obviously, it's going to be surveilling, it's going to be surveilling staff. It's all also going to make, you know, visibility, things about customers too. So a very, very good point. There is a big brother element. When you say is it too dangerous? These, these products are happening and they're out there. So it's a matter of grappling with them and regulators should be thinking about this. This is this is really, really good points. Thank you. And then someone said, it's good to emphasize, being careful about chat to BT, because it produces very credible information, looks totally credible, but it may be complete nonsense. So a very good warning there. Next, the next point, please. Now, what I'm on about here is this is just the last example of how how it may affect AI, may affect evaluation. A lot of evaluation and quality assurance relies on documentation. You know, if you think, if you think broadly about evaluation, often people will use documentation to assess the quality of a program or to screen which programs need further investigation. So at the bottom we've got what the program is actually doing, then we've got the documentation, then we've got the assessment of the documentation. So really, that the issue here is AI could produce almost perfect documentation for every program. This will disrupt the way in which documentation is used to screen programs. If you see what I'm getting at there, basically, say you take it even if say you think about food safety system as an evaluative process. So look at everybody's documentation of all the companies producing food and if there's that someone's documentation is weak, they'd say that we need, they need to be revved up. But if everybody's producing overnight and just ask chatGPT to produce perfect documentation, then that kind of evaluative plank gets knocked out of the process. So I think this is a space to watch. Very interesting development. Next slide please. Just to illustrate what I'm talking about here, this is actually from the EPA Environmental Protection Agency in the States. This is, I know you can't read it, but this is a set of guidance for how you evaluate your documentation. So this really just highlights the fact, this is the Environmental Protection Agency monitoring what people are doing. And they've actually got guidance for how people can, you know, what needs to be in people's documentation at this level. So it just shows the role of documentation in evaluative and quality assurance processes, which we need to keep our eye on. And then we've got documentation is only useful if it's used and implemented properly. That's absolutely right. Thank you very much for the comment. But the thing is that it costs a vast amount of money to work out whether people actually are, are consistent with their documentation. That's really what the evaluators get often paid to do. But anyway, the main point here is that a lot of people's documentation is going to be a lot better than it has been in the past. And how will that disrupt the evaluative and the quality assurance process? It's just a little thing I want to raise. Next slide please. Okay, on outcomes evaluation, we have the poll and we've got, obviously, possibilities for disruption there. Next slide, please. Thank you. That one. Yes. So this is from the Harvard Business Review. So this is a bit using AI to track our customers feel in real time. So there now, so this is, you know, the sound of outcomes. So they're really saying everything's moving. We can now start monitoring real time outcomes. So I really just wanted to put that up to illustrate that point, but also to illustrate a further point if you go to the next slide. And what they say in this next slide is basically they say it's easy to see why quantitative surveys became popular because you could ask a lot of people. But qualitative research was too labor intensive, which is what they say. But now they say technology is changing that whole thing. And they're now suggesting that you should go after qualitative information, actually they're saying first as much as you do quantitative information. So I just wanted to highlight this. You know that anyone who's who is I don't know, I'm not sure it's actually sure whether but you know the qualitative quantitative debates and evaluation rage like some sort of war for a long time there when I was doing my PhD. But basically this is saying that AI changes the balance within quantitative and qualitative makes qualitative a lot easier, a lot less less expensive. So I think there's just sort of fascinating things about methodology and mixed methods which are going to come out of this AI revolution. So that was the highlight that next slide, please, Marie. OK, so now to the third point, I've just got another five minutes or so, Marie, on social implications, social impact of AI for evaluators. Now, this is a huge topic. We could we could run a whole day workshop just on this. I've just written. So I've just really referred to my book, trending AI, 30 fresh ideas to help you think about artificial intelligence. So this really what I'm trying to do in this book is talk about a lot of the impacts, including the social impacts and obviously they some of the books not about evaluators, but that they are relevant. If you could flick to the next slide. Thanks, Marie. So that's the 30 fresh ideas to help you. That's just what the book looks like coming up on Amazon in September. So just next page, please. OK, so these are just I don't have time to talk about these in detail, but these are just these are just some of the social impact concepts on the book. One is this idea of upskilling AI. And if you look at the page I put on the resource site, you'll see the best way to think about AI is that it started off that we think about chat box started off as just a knowledge function that could tie in and receive Britain. And it's slowly been getting new skills over time. If you look at that web page which is up on the resource page, you'll see you can just see basically AI is just getting more and more skills. It's able to is able to talk is able to hear is able to write software is able to use software sooner will be able to use money. It can interact with websites. So that's just one way of looking at AI progressive process of upskilling over time. Next one is analysability, which is the idea which we really just explored AI's ability to analyse all sorts of data very, very efficiently and and we just talked about the qualitative that can analyze qualitative like this qualitative aspects of that data. Next concept is knowledge ability. And what this is getting at is AI changes the whole dynamic of knowledge accumulation, knowledge sharing and knowledge transmission. I don't have time to go into it here, but basically there's a whole lot of interesting possibilities for around educational inequality where AI all of a sudden AI provides a lot of people with possible access to all sorts of knowledge they didn't have access to in the past. Nudge ability is another concept I talked about. That's the idea we all know there's algorithms influencing us and the idea of nudge ability is for us to get control of those algorithms and get them to nudge us in a positive direction. Rather than us on social media just being nudge to buy stuff nudge ability is the idea that if we got control of the algorithms they could nudge us in the directions we want to get and we want to go over. Trustability is another one. So in an environment of AI where there's endless what I'm calling info trash anyone who can give a trustability network you know say you can trust this information is going to be in a really good position. Next one is revenge of the real and that's the idea that people may rebel against all of this this AI and artificial world. You can see that almost in that in that mission impossible film that came out the the star of the actually he wrote a motive with a motorbike. Yeah, he wrote a motorbike off a cliff. He actually did do that. He had a parachute. That was a selling point that wasn't just created by it wasn't created by artificial intelligence. He actually did that in the film. That's kind of example revenge the real people actually wanting some real stuff not just artificially created stuff. And then last the AI anxiety that's just what I want to finish on if you just go to the next slide, please. OK, so as a psychologist also clinical psychologist so I feel obliged to deal with the help people deal with AI anxiety. So I've got a little acronym there. So this is really what can happen when you start thinking about a and I do know a lot of people who invest in the cutting edge. They actually they sort of are staggering around some days about where they see where things are going. The first of all is to check out AI as much as possible if you know about something that can reduce your anxiety about. The next is just reflect on what it means to be uniquely human. You know, we're people are worried about AI taking over. But there are a lot of human aspects altruism, compassion, certain aspects of mindfulness which are uniquely human and we should we should start to value those. So don't don't really think AI is going to replace everything about being human. Next point is the eye. If there are major impacts say on jobs, it's not just an individual problem. It's a social problem. So don't personalize it too much. If say 30% of the population is put out of work because of AI, it's going to be everybody's problem, not just your problem. Next is limit your anxiety using standard kind of anxiety minimization techniques as you talk to a psychologist about. And then lastly, the last hours show leadership and responding to AI. The best way of getting on top of worrying about something is to get actively involved in pushing forward on that thing. And and just what the people really said in the comments really thinking thoughtfully about it and getting involved in discussing and discussing AI in various ways. Thank you. And just onto the last slide. OK, so if any of you want to be notified when my book comes out, just email me at Dr. Paul W. Digan at Gmail. I'll throw that up in the chat in a minute. If you want to get in touch, I'm on LinkedIn. Just go to search LinkedIn. Dr. Paul W. Digan on just search on Google. You can go. You can check me on my website. So if you go to URL Dr. Paul dot online, you'll be able to be able to find my website. And then lastly, the notes from this this presentation are called I can dot consulting slash P 50. So there's some, they're not the notes. They're more the resources for this. So just before I finish, I'll just check. So we haven't got any more comments at the moment, which is cool. I'll throw up my email address. So if you want to get in touch about anything, just email me at that address. But if you want to give in touch, if you want to know when books coming out, let me know. And I'll hand back to Marie. So thank you very much. Sorry, there's a very, there's a lot of stuff to cover, but I really just wanted to give you a lot of stuff so you could think about it. And then lastly, Marie, I'll just post up a couple of questions for people could email about me for the feedback. So thank you so much, Paul. That was, there was an extremely in-depth and illuminating presentation. I would just stop the share right now so we can all, if you feel like it, do turn on your cameras. And even though we have 71 people, please put up your hand and we can take questions. Just to start it off with Paul, I have a question. I've been playing around with AI, not on a professional level, but on a personal level for quite a few months as soon as it has come up. And interestingly, I do note that the quality of the results also depends on the wording of your questions, the quality of the question, and also if you use chat GTP or Googlebot, and I haven't played around with Bing as yet, but I'm just keen to get your views and probably everyone's views on what they actually prefer in terms of plain English, GTP or bot or Bing, and what they have noticed like, I'm keen to get everyone's views on your personal use of these tools in terms of how you phrase a question, you know, the detail, the level of detail you use when you phrase a question. So yeah, the flow is open to anyone who wants to talk to this point. Paul, what do you think? Well, I have heard some people comparing chat GTP, Bing and Bard, and they thought, well, it depends on the nature of the question. If you want any fresh information, you've got to go to Bing or Bard at the moment, you can't get it from chat GTP, because it's not hooked up to the internet. Bing sort of is fairly limited. This is just very impressionistic, depends on the question, but it kind of tells you, it'll tell you answers. Bard, which is a Google one, can give quite more detailed answers. So I had some of the other day were comparing responses on the three, and they thought for that comparison, Bard was really good. But it's like anything, depends on how you want to actually use it. But back to your question, there's this thing called prompt engineering. It's about the way you ask the question, and that's really important. So you should really play around with the way you ask the question and have it online, ongoing dialogue with it. Absolutely. I'm just checking the chat. And there is a question here. But do you have a concern about the output from chat GTP is generally bland anodyne? Can you start to spot AI output because it is usually verbose and boilerplate? Do you think this will improve? Yeah, that's absolutely true. So I think the deficits of AI are a good thing. So a lot of people say they put in all these bad things. I think those things are good at the moment, because the central problems of society is trying to get our heads around this as quickly as possible. So the more problems AI has at the moment, I think the better if you get what I mean. Imagine if it was absolutely perfect, that issue would be very dangerous situation. So just one minor point there. When AI has a problem, I should go, yay, that's good, because that gives a little bit more time. You can use prompting. So you can say, write it in a different way, write it in a more exciting way. And to some extent it does that. But if you don't do that, it will produce a beauty plan, kind of boring-ish response. So, and I think for those of us who, I think really the added value will be trying to do things a little bit differently. So it does produce a boring response. The way you ask the question, although you can modify that to some extent, but I think still people will be able to have a bit of an edge in terms of response of this, hopefully for a while. Thank you. Do we have any more questions? People, feel free to unmute yourself. So I'll confess that I haven't been experimenting yet, but I should and I'm feeling like a dinosaur and I need to get on with it. But do you have any guidance on how to learn good prompting other than just experimenting? Is there, I've got your six rules, you've given us a few things that I'm looking forward to looking at your website. If you have a look up on the internet, you'll find lots of discussions of prompts. The best thing though is to play around with it and see what you can do. But you can just remember, you can tell it to play a role. So it's quite useful to go acting as a so-and-so. Because if you just ask it generally, it doesn't know what it is. Like if someone asks an evaluator or a psychologist or something, you already know, you respond as an evaluator or a psychologist but poor old GPT doesn't know what it is until you tell it what it is. So it's always useful to start as an evaluator, as a psychologist, as an expert this or an expert that, or as a primary school teacher too. Because I mean, I'd say just play with it basically. You can have a wee look at the prompting stuff on the internet. But I mean, you can say to stuff like I said the other day, explain nuclear physics to a five-year-old child. And it started talking in a way that a five-year-old child could understand. So play around with it. But I would, if you can get hold of the paid version, there's gonna be a little bit smarter than the other version, the free version. I was sneaking another question. So Paul, right now there are many organizations that are writing official AI policies. You know, how people should kind of be cautious when they're using. And as these things are still in development, many employees do not know the extent to which they are allowed to use AI for client-centered work. Do you have any recommendations or advice for such organizations that are currently implementing these policies? Yeah, I think it's a terrible model. I think organizations need to get on top of it. They're not necessarily banned the use, but they need to be very clear. Because what's happening at, what I suspect is happening at the moment. People are randomly using it and organizations don't know that they are. So say you see to a government agency, how much of your material is produced by ChatGPT or another large language model? I suspect government agencies would have no idea because analysts might be feeding that up to their supervisors. Whole evaluation reports or policy analysis might be going to supervisors who don't know it's produced by ChatGPT. And then it's being put out on the website of the organization. So I think organizations, I'm not saying they should ban it. They just need to be very clear. They should be providing guidance as soon as possible to their staff about it. And the problem is the timeframe is extremely tight. This is, you know, like social media took 20 years to get up and running on the internet to 20, 15, 20 years. This thing's happening over months. So I think that they should be very proactive. They should just call it interim guidance. And they could say, this is what we're doing at the moment. This is how you can use it. So they might say, at the moment you can use it to have a discussion with about concepts, but don't feed any AI written text into the system unless you clearly identify that all the way through. You know, that's the kind of thing. And then as they don't put confidential stuff in while we work out ways of having this firewall AI environment. So if I was an organization, I'd really get guidance. I mean, the DIA has already put out under Paul James put out some guidance at the moment for the public sector in New Zealand. So that the sooner there's guidance the better. That is a brief comment. Marie, yesterday I attended an Institute of Directors deep dive on AI and governance. And there was some talk, I mean, obviously from a governance perspective. So I was thinking about the legal implications, privacy issues, those sorts of things and what directors or governors might be needing to think about these sorts of things. But yeah, so that was an actual and like a deep dive session. I found it quite useful. So it might be the sort of thing if anyone's on the call who wants a bit more and another alternative avenue, just from a different perspective, it might be something to keep an eye on because it's certainly an issue that the IOD is taking some time to share seminars a bit like this with. But yeah, that lack of a framework is one of the key things. And my takeaway message was actually you need to come back to, well, what are the core underpinning values of the organization itself? And then you need to develop your operating principles and alignment with that. Some commercial organizations are going to be less concerned with certain things over and above just meeting kind of legal requirements. Whereas others might take a particular position on what is or isn't acceptable because it is or isn't in alignment with their organization's values. Yeah, it's just a little reflection from a very fresh experience. Thank you very much, Kara. Yeah, Paul, would you like to respond to that in some way? That's fine, yes. I think totally, the thing about why AI are fascinating. Everywhere you look, there's all sorts of issues. I was at a meeting at a conference yesterday about AI. And just this whole thing of liability which obviously relates to directors. Someone was saying, directors, because of health and safety requirements and that requirement being on directors, that's changed health and safety. Now, the issue with AI is who's liable? Say AI makes a mistake. Is it the directors of the company which they would be involved? But is it the people who develop the AI? Is it the material the AI is trained on? And AI doesn't have personal liability the way that humans do, but should this be the case? So you just even look and just you've just raised a tiny little bit and it's like a hundred years of issues. It's a little bit like that, little baby. And then it's like, how do things people feeling from a psychology, how do people feeling about themselves? Another one. And then it's like the whole society is permeated with massive uncertainty about everybody's future. So everywhere I look is just boom, boom, boom, big stuff. So basically exciting, but kind of frightening at the same time. Even though we are at time, Ruth has a hand up. So go for it, Ruth. I just had a quick question about, you talked about the potential to not just sort of scan a sample of reports that a human could do, but to scan everything a program has generated or whatever. But that is putting data into or through a system. Are there some providers that you know of that have put more emphasis on being locked down or data staying within Australia or within New Zealand or giving more assurance about how the data will be managed? Well, this is exactly what has been worked on at this moment. So as I understand it, Microsoft, Azure will be trying to, will this or will be offering that sort of thing. Or yeah, and they're not the only ones. There's a company in Wellington called Hoist, which is working on their products called Hoist. It's called Endgame. They're trying to do that. So everybody's, everybody, the basic thing about AI is if there's a, people say, oh, this is a problem, that's a problem. If there is a problem, you can be sure that like a thousand people of PhDs are trying to fix that problem because they'll all get millions of dollars each if they do. So pretty much if AI has a problem, someone's working on it as we speak. So that is an identified problem and there are solutions. I just, already there may be, there are solutions emerging and they will more and more. But the obviously people to talk, who would be like government IT people whose job it is to protect data. But I think that that problem is going to be sorted out. It'll be secure AI environments very soon. Thank you so much, Paul, for a very practical and engaging session. Everyone, this recording will be uploaded in four weeks and you will get, I'll ensure you'll get a prompt for that. And thank you. Thank you very much, Paul. We can't wait to read your book when it's out. And I'll make sure everyone gets knows about it when it's available on Amazon. And good luck in your AI endeavors.