 Hello everyone and welcome to a talent development webinar from dbrown consulting I'm David Brown your host and I'm very glad to have you here. I'm sorry for this late start. All right great. So Today we're gonna talk about Training evaluation. That's our topic for the day. So well your speaker today is David. That's me David Brown I've been doing this for like 30 something years training and Developing courses talent developments have been involved in the industry for quite a while I'm also an ATD master trainer and ATD master instructional designer So that's me and I do a lot of technical stuff like building financial models and stuff But also of course designing training. That's I think I like that more So about the sponsors dbrown consulting are the sponsors for this webinar. It's dbrown consulting We do training consulting and payroll Training bit is more analysts. We're the analyst trainers. We train analysts financial analysts reporting analysts data analysts any analyst If you go over to office training hub.com, you'll see some of our online e-learning courses Where we are the analyst trainers And consulting wise we build financial models and we'll build bi solutions for deals using power bi And we have a payroll package which guarantees confidentiality Because we know that confidentiality is the most important thing for any payroll And of course calculating the accurate Accurate taxes and accurate salaries is correct. It's important, but confidentiality is really key So we have a service that guarantees that We have our courses more as I said analyst courses. That's what we do mostly So thus we have financial analyst financial modeling Business intelligence excel and then we have a bunch of online courses as well And we have affiliates financial financial modeling institute Microsoft and ATD those are our chorus affiliates And for those that like technical stuff, you would like to join our meetup groups. We have three meetup groups Meetup groups for financial modeling meetup groups for power bi Meetup groups for excel So if you join those meetup groups, you'll be able to get a notification and invites to our live meetups and our online webinars so this topic is mainly Kind of started off by don uh, donald kirk patrick. Yeah And he is the don so to say of evaluation He developed four levels of evaluation And that has been the gold standard for evaluation ever since I think developed that in the 50s I mean in pretty long time ago He's late now. He passed away. Unfortunately on 2014, but at least he He had a really wonderful life and he's contributed extremely heavily to the training and development and talent and development industry He was once the president of ASTD. That's currently called ATD Association for talent development And he's written so many books management and training and he's really Positively affected the training industry. I mean he his name will live on for a very very very long time So here we're going to talk about those three levels and we're also actually going to talk about what they call the new world kirk patrick model which is a slight modification to the old model So it's quite A nice thing we're going to talk about there as in the new the new additions to the four levels So if you already know the four levels they have to know whether you know the additions to the four levels so Let's get started. There are really four levels of evaluation And we're going to discuss these four levels and we have some comments there and see how everybody Uses those four levels. So the number one level if anyone knows what it is type in the chat. What is level one? What is level one? Who knows type in the chat? So we're going to start with level one. So what's level one? Who knows level ones? They're four levels level one level two level three level four. What is the first level of evaluation? People don't want to just type type type Okay, nobody wants to type just guess if you can't just guess okay So the first level of evaluation is reaction So first thing you need to evaluate is reaction now I I think nearly every training in the world or when they say oh, do you evaluate training? They say yeah, yeah, we evaluate training and what they really do is just level one Level one. There's level one level two level three level four But I know most people when you ask them do you evaluate training they say yes And it's actually just level one reaction. What did we mean by reaction? You remember those? Forms you give people at the end of the training and say hey, please fill this form fill this evaluation form And then you say did you how well did you understand what was thought very well? So not so well Or not satisfactory. How how well did the Let's give you time to practice all those typical questions are basically measuring reaction, right? So you're measuring customer satisfaction and your customer being the participant, right? Yeah, so that's that's what was originally defined in kappatrix model reaction So in the new world kappatrix model the new world edition So if you go on to their website, you'll see the new world edition is engagement So it's not just reaction. We're measuring we want to also understand the degree to which participants are actively involved in in and contributing to the learning experience So we also have to measure engagement. So part of level one is reaction, but add to it engagement And then also relevance. That's another thing which they've added to level one. So we're measuring not just reaction But engagement and relevance So the degree to which training participants will have the opportunity to use Or apply what they've learned on training. I mean in the training on their job So that is the relevance. So You could do a training that's absolutely not relevant to your job. So that's You should measure that in part of the reaction. So level one is reaction Yes, but you should also see look out for engagement and relevance as part of level one So how many of us here do level one? Just say yes or no Yeah, so relevance is key and I think is one of the most Um Most used levels of evaluation. Yeah And Reaction really. Yeah reaction. And then we've talked about engagement and relevance being the new world additions to the Level one. So let me just type that in the chat. So you remember so it's engagement and Reaction Oh, no, so no reaction relevance relevance being the new additions So Great, so what is level two? If if level one is reaction, which is what we do every time we just put Prints out to those forms and people fill the forms and says yes, the training was very nice. Excellent. The food was good What is level two? Someone should type level two So what is level two? Come on type please type type type What is level two? I'm waiting for you guys. You have to participate. You have to Engage I need to get you guys engaged. Yes. And is this relevant to you? I'm doing my own level one evaluation right now. So Can you tell me what is? level two Okay Nobody wants to tell me what level two is. Okay. Let's just move on to level two So level two when it comes for evaluation is learning So level two is learning Yes, the whole idea of training is learning But really has knowledge been passed across so when it comes to learning What are you really asking yourself is You know, whenever you train right and this is this the old car Patrick model, which I still use a lot Whenever you are training or if you are a trainer or your facilitator You are trying to affect three things three main things. You're trying to affect either skill Knowledge or attitude or better put knowledge skill and attitude in that order. Okay So when you're measuring evaluating you're evaluating whether or not you've affected knowledge whether knowledge has passed Skill has passed and an attitude has changed But really level two if I want to define it properly, let me give an example when you say knowledge You mean you know something I know it. So I after the training do I know this thing? Yeah, I know it. So knowledge has passed across skill is If you can actually implement or do it you can do it right then and then in that training You were actually able to do it right? I can do it right now kind of stuff Then what about attitude? I believe this will be worthwhile to my job That's you you have an attitude that says this is going to be useful for me So this is still during the training, right? So you know it you can do it and you believe that it will be worthwhile for you On the job. So that's your knowledge skill and attitude But the new world additions to this Is confidence and commitment. Those are the two new additions to the level two learning So the new world chiropractic model added confidence and commitments to these learning level two So confidence basically means I think I can do it on the job. So I think it can do it So the attitude was I believe this will be worthwhile Of course, I think I can do it then commitment is I intend to do it So how do you measure that intention for people that people intend to do it? So that's also a part of level two learning. Yeah, so how many of us do level two? Actually measuring knowledge skill and attitude So let me tell you how we do it when we do our trainings During the training because this is actually measured during the training itself How can you measure knowledge skill and attitude during the training? So for us, we do a lot of technical courses. So for example, we do excel we do power points we do visualization And stuff. So when we're doing that training what we have is we have a clicker We have clickers all participants have a clicker And that clicker is what they're going to use to answer very short quizzes during the training So let's say two hours pass or four hours pass There's a a quiz comes up and they ask you ask a question with the abcd option And you're supposed to use your clicker to click an answer So obviously if you get the answer right, it seems that it means you know it And another thing we do in our trainings is you actually practice what we teach since it's technical And most times we're using a computer. They actually have to practice They have to go there and type go there and answer write this formula and actually do it. So yes, we can measure skill But attitude I believe this will be worthwhile to do on the job The way we measure that is really a discussion to say, okay How are you going to apply this on the job? You've done this How are you going to apply it and you have a general discussion? It could be a pair to pair Just in simple pair to pair and then you can now Give give the general group a feedback a feedback loop where I say, yeah, yeah This is how I'm going to use it on the job Then you now reach the confidence level because of that pair to pair There's more confidence and hopefully there'll be a commitment. So why how you measure how you do that commitment? So how you ensure that commitment is getting people to actually write it down Writing it down or even emailing each other committing to do that new learning or to use that new learning So that's level two. It is still on the in the training. You haven't left the training yet Yes, still this is level two measurements. So how many of us do that? So the learning, um, I don't know how much of that you heard I was talking about learning being During the training yet checking whether knowledge skill and attitude has has been affected by the training And I said that we use clicker technology So we have a clicker every participant gets and the answer questions during the training So we understand whether they're assimilating is extremely important because when you ask we would do you understand? And they say yes. I mean that's not how to ask Actually, give them a quiz just give them a very short quiz And there's so many other techniques to to help you Measure whether knowledge skill has and skill has has been affected and even attitude So what you could do is you could do word puzzles You could do a very simple word puzzle and ask them to fill out the puzzle based on all the things You've taught them that day. That's a really really cool easy way to to measure Knowledge skill and attitude during the training and then what the new world additions were were confidence and commitment So let's go to level three Level three is behavior So level three is behavior now behavior Is a very key aspect and how do you measure behavior? Right, how do you measure behavior? Behavior typically is the degree to which Participants apply what they've learned during the training when they are back I mean they're applying what they've learned during the training when they're back on the Job when they're back at the desk have how how well do they apply it? What's the degree of application? Right, so behavior How how would you ensure that behavior is measured? So obviously there was behavior before the training And then maybe a week or two weeks after the training there's behavior after I can give you a classic example. I had a Client we went to a client and kind of after a course a major course was done I just to ask about how Participants were doing are they using the skills? And then we got a Response that nope, they're not using the skills They learned it when they came back. They were excited But when they were now asked like a month later to exhibit that skill or to to use that skill to do something they couldn't And they were not really happy. They've spent all this money. That doesn't this doesn't seem to be a return on investment And what happened is we've built behavior into our design But the issue was The hadn't done that behavior piece. So what we did is we We have an assignment We put an assignment at the end of the course and said look you need to do this assignment Within a week of completing this course or you're going to forget about everything And really that's what it is. So most times you forget. So that assignment The only thing that we didn't do as well was monitor them or talk to the client and say hey Your people have an assignment. Please. Could you monitor them to make sure they're done within a week? And your key buy-in will be from their supervisor You need to be able to talk to their supervisor and then make sure Or or kind of sit to the supervisor and let him understand why level three is very important And you need to change behavior and the only way to change behavior Is if people are allowed to or encouraged to apply what they've learned on the job And how you facilitate that is two ways you could give an assignment And then you could also give a post training support post training the support mostly in the sense of maybe like An online maybe Small refresher So there are online tools online training tools you could give and then make sure you follow up to ensure the actually do it Because no point given doing some Tools and people are not doing it. So so that's how you affect behavior Can anyone tell me another methodology you use to ensure that level three happens? And when you evaluate behavior, you could actually see behavior change anyone Care to share The new world edition, you know, I said this is the new Kirkpatrick model So what have what have they added to this Kirkpatrick model of behavior? Well, they basically added drivers So they basically so what are the drivers or the other processes and systems that actually reinforce or encourage Or even reward performance of this critical behavior to change you're trying to look for on the job, right? So so you're looking for changes in behavior on the job You need to basically have systems in place processes in place That will reinforce and encourage that change So that's all part of the level three behavior Okay, so we're moving to the last The last level of evaluation and what was that last level? The last level is results So level is results. The last level is do you measure results? How well do you measure results? So the degree to which targeted outcomes are called of training Accountability package. So that's generally what results are She could even call it a bit of return on investments, right? So we're measuring Results now who does this measurement? How do you really measure results itself in practical terms? You can say that The training has resulted in a 50% increase in something or Or the training has resulted in 20% savings in time anybody here Okay, so Lawrence uses appraisals. So use appraisals to measure results And and then of course when maybe if I'm getting you right somebody scored maybe evaluation was 60% and after the training evaluation went up to 75% because performance went up And you're now attribute. The only issue with that is attribution. How are we going to attribute that performance gain to the training? Because a lot of other things are also affecting performance. Who knows the guy could have probably Been trying to look for a house for six good months He's homeless and he finally got a nice house and now he's relaxed and now his performance has increased And we think it's actually the training he did two months ago So it's it's attribution. It becomes an issue right or Lawrence. Do you find that issue as attribution It would be nice if you complete your mic up Lawrence I would like to contribute like you to contribute Because I don't just like myself talking every time So if you could request for a mic for to speak I can then release your Your mic so you can speak I'd like to hear from Lawrence if it's possible all right Okay, so that is the results so results we understand results But then there as I said, there's a new world addition. So there's the new world Kirkpatrick model and the new world Kirkpatrick model adds a leading Indicators to the results Level four so level fortunately. Yes, you should have results return on investment, but you should also calculate leading indicators Yeah, so short-term observations and measurements suggesting that critical behavior are on track or or to create a positive impact on desired results so Quick observations and then those measurements that actually tell you that things have changed So this leads to return on investments. Really. That's what results are And and you should be able to actually calculate return on investments But let's give a little bit of credit to Kirkpatrick for this model that he developed and our ATD in 2000 I believe is it 20 2003 I can't really remember when exactly they paid attributes to Um to Kirkpatrick and that was just probably a year or two before he passed away So let me play that short couple of minutes tribute for you Before we begin today's program I'd like to recognize Don Kirkpatrick a legend in our field With 2011 marking the final year of regular speaking engagements This ASD international conference and exposition is the last opportunity for all of us to hear from Don We interviewed Don about the origin of the four levels of evaluation And what he thinks is critical for the future of the field Here's Don You know, that's about the most common question I'm asked. How did you come up with the four levels? And I tell you it's a long story But I was teaching at the management institute the university of wisconsin And I had my master's degree from the school of business. I decided to go on and get my phd And in doing that I decided I'm going to do evaluating the program I was teaching in between then and nine that was in 1954 When I got my phd, I had gone to the school of education And evaluated the programs basically one and two but I had said something I said The day of reckoning probably is going to come when top management is going to look for more than level one and level two Which are reaction learning So then between 54 and 59 I did some research on behavior and results. I went into companies I found out are you using what you learned? And if so, what can you show any evidence of productivity or quality or more sales or anything from it? So I did some research and then in 1959 Bob Craig editor in the astd journal called me and said don I understand you've done some research on evaluation Would you write an article? And I said Bob, I'll tell you what I'll do. I'll write four articles One on reaction one on learning one on behavior and one on results He said we've never published a series before and I said it's a good time to start So the value of that book is not just my four articles the value of that book is the case studies we have for Toyota and from caterpillar and from IBM and from a lot of other companies And people have been able to look at those case studies and say well, here's a reaction seat I'm going to use or here's how I'm going to measure learning or here's what I can do to measure behavior Here's what I can do to measure results And that is one of the reasons why it's been so popular throughout the world is because the case studies are not just my My concepts and theories and principles and techniques So I would suggest the real key to the future is to build this partnership with managers Get them on board in one of our books implementing the four levels I've written a chapter on how to get managers on board And that's what you need and you need top management involved as a partnership In terms of just it isn't just training that by itself it's doing it It's the things that they want you to accomplish and then you know that ahead of time And then you know what your objectives can be and I think that's the trend in the future And if you want to be successful in the training profession, let me suggest you take that advice about get managers on board And starting with them saying if you want this program what you would want us to accomplish And what are your expectations and what does success look like to you? Don on behalf of ASTD and everyone here today Thank you for your service to the association Your colleagues and organizations worldwide You have helped learning professionals everywhere understand the importance And the impact of measuring our work Don please stand so we can recognize you for your work Great we're back and so that was excellent. That's just a tribute to Don And so what did we learn from that any key points so we can take out of that? Can we just type in the chat? Any key points we can take out of that short talk by Donald Kirkpatrick Well one one I got was get managers involved Get them on board Forget those managers on board get them get your supervisors on board because if you do a training you go back and your supervisor is not on board you can Your return on investments is just going to go down the drain so Yeah, exactly. Lauren's got that as well get the managers on board What else did we get? Another key thing you should always ask if you if you're designing a course and I'm an instructional designer actually an atd master instructional designer and the Easiest question you can ask a client That will help you understand whether or not the training will really make sense is what does success look like to you? That's a key question. What does success look like to you? So just ask your manager that or ask your md that when you say no, I need this train Okay, what does success look like to you? Then write down everything that person says and then ask yourself So if this is what success looks like what can I do to make that happen? What changes do I need to make and what are we going to put in the training that would affect that change? And that will give us those four levels that we're talking about Right. So what does success look like to you? So for us here as in what I want to do is just ask us a little question Let me do a quick poll. I'm just evaluating myself here a bit Let's let me ask you this question. So I'm asking So where are you based on what you've learned so far? Do you know it? Can you do it? Is it worthwhile? Do you think you can do it on the job? Or do you intend to do it on the job? So let's answer those That's just what you've learned today, right? Just what you've just learned, right? Based on what you've learned so far So intending to do it on the job that takes a while really so I can agree that no one's answering They are saying they intend to do it on the job Uh, I think I can do it on the job Well, again, maybe level three level four is very tough for some people But level one and two I think you can actually do level one and two Level three and four needs a little bit more planning All right, so I'm going to end the poll Right. So here I want to talk about Return on investment Right return on investment And if you look at the screen, I have a book by Um, Jack Phillips is also one of the top dawns and return on investment is very well known for return on investment calculation in the industry And ron drew ron drew ron drew stone those are the two that wrote this book How to measure training results. It's actually a very valuable book Is a practical guide to tracking the six key indicators So they have six key indicators of the foot in and well the six key indicators Is just another way of like the four levels, okay? So, you know, there are different models for everything. There's adi the sam all of them see Practically the same thing, right? So there are six indicators So First of all, they has a 10 stage process for measurements. That's what they have. They have a 10 stage process So I'm going to just run through these 10 stages for you. You could write them down But there's 10 stage process for measuring return on investment And then I'll just do a very quick demo on maybe on excel of a return on investment calculation A very simple return on investment calculation that you could use Before going into the complex ones. So what are those 10 stages or to process? 10 stage process for measuring or for measurement of return on investment. Let's have a look at that Okay, I can see Lawrence's comments very true that It seems only Lawrence is commenting everybody else is quiet. Why anyway Level four was also a bit difficult to achieve And he's thinking it's because the managers didn't get the managers on board And usually that's what happens if the managers are not on board the supervisors are not on board The co is not on board things will not change all right Okay, so that's a good one. So let me quickly go through these 10 so we have Develop training objectives as number one So if you really want to do calculate return on investments The first thing you need to do is really develop training objectives And your training objectives must follow All we call blooms taxonomy if you know blooms if you don't know just check him out Just go online for blooms Uh when he comes to creating objectives and You have to create objectives at each level. So that's level one level two level three level four You must create objectives for all those four levels Now jack phillips He is on um his own model has five levels So his five five levels are reaction learning application impact and return on investment So of course if you remember karp-patrick's levels were Reaction learning behavior and results. So here we have reaction learning application, which is behavior Impact, which I guess is results and then return on investment. So everyone kind of builds their own little levels, right? So how do you when you develop training objectives? The basis for the measurement really is at the end of the day or a business So most of us do corporate training you need to increase revenue or reduce cost your training You should be able to track whether or not your training increased revenue or reduced cost And if you can track that then you're not doing return on investment calculations And the best way to track is to write very clear-cut objectives Write objectives for reaction for learning. So example for an objective for reaction Is you need to define a specific level of satisfaction and reaction to the training as it relates to the participants And as delivered to the participants. So we all do that. We do reaction. We do learning Divine defining a specific knowledge or skill to be developed. That's the learning right the application on the other hand Is you need to define behavior the most change So what behavior must change that's the application the impact is to define the specific business metric The specific business measure that needs to change So you you find out what that metric is before and then after the training you measure it again That way you'll be able to calculate good return on investment so the next The next objective or the next In the framework we're talking about Not care Patrick now we're talking about jack phillips 10 stage process of measurement So the next thing is develop evaluation plans and baseline data So you need obviously a starting point. What's my starting point? What's current state analysis before the training happens? And you as an training expert need to sit down with your manager or sit down with your client And draft what exactly your baseline data is so hard data categories you mean you could include things like I don't know How much waste do you currently generate? Maybe you're you're Trying to reduce paper usage or something. How do you effectively communicate or how do you effectively review without using paper? So okay, how much waste do you currently have right and that's quality and then the output as well What's what's your production like how much work is currently being done? So well you have you just have certain metrics that you write down and that becomes your base data, right? So that that's critical else. You not know what you're measuring to You can't say I improved. How would you improve? So stuff like Just list out factors when developing an evaluation location duration participants involvement management interests. Those are just some factors So Next thing you should do is collect that data During the training. So what we do is that clicker clicker technology You collect all that data how people are doing during the training once we do We like to do all that data collection. We do the measurement before And then during the training we now gather that data you can use excel to analyze it Or if you use power bi to analyze it or you collect the data nonetheless Then after collecting the data You have to collect the data again after the training. So you've collected during you've collected before You collected during you also need to collect after So that's the key part really after the training. How do you collect the data now? If you're a consultant, obviously you need to go back To the client or ask the client to collect the data and give you the data so you can analyze Or if you're working there, you don't just finish the training and leave you need to go back to the managers Ask them to get that data, but you'd have planned all this in the beginning before you even start the training So collect the data after the training. So follow-up surveys. For example, you could use Observations on the job. You could use that that's data collection Yeah, you could do interviews follow-up interviews with the manager with them themselves You could even have focus groups to collect the data. All that is how you collect data. Those are methodologies for collecting data Yeah, you could even give an assignment like we like to do we give an assignment related to the training and say Hey, you must finish this assignment by so so date complete The market and then we rate it to give them with actual actual rating So all that is data collection after the training. All right. So that's number four, you know, there's a 10 stage process So, okay, I've just given some ideas here All right, so then Okay, yeah, we were actually supposed to have a guest speaker today, but unfortunately he couldn't make it So that's why I've just prolonged my talk usually every month we have a guest speaker and then The guest speaker we at least talk about their own experiences and what we're talking about So isolating the effect of training is the next That's stage five in our 10 stage process, right? So stage five is Stage five is isolating the effect of training again to be able to do that. You need to Probably have a control group. So you have people that did not do the training And then the people that did the training. So whatever data you're collecting You have those that didn't do the training. How well are they doing after the training? Those that did the training. How well are they doing after the training? So those are two things you need to to do that's an example of how you can isolate the effects of training There are many other ways, but um, that's just an example The next stage again Okay, let me just leave this slide for you. You could have a look at some other examples To forecasting is just you're using a predictive model Yeah, so various predictive model in change in variables and stuff like that But control groups are really, uh, I think pretty pretty effective All right. So the next thing is you convert data to monetary value How do you convert data to monetary value? So a simple way which I'm going to show you in the example is Well How long were you doing this task for? I was doing this task. It took me two and a half hours to do this task after the training One month after the training, how long are you doing this task for? Oh, it's taking me one hour So you've saved one and a half hours. Maybe you save one and a half hours every week So if you save one and a half hours every week and you do 50 weeks in the year So one and a half hours times 50 gives you 75 75 hours. How much is an hour worth? So in five times, whatever the hour is worth is technically your savings. That's how much you've saved So that's just an example. There are many others So convert data to monetary value. That's step number six if I remember. Yep. That's step number six Yeah So what is step number seven step number seven is Identify the cost of training. How much was the training? So identify the cost of training Well, what is the total cost instructor fees if you're doing an internal training You need to put a value to the instructor's time because usually the instructor is a colleague or somebody like that Maybe somebody traveled from somewhere. There's a hotel cost And then there's also opportunity cost, you know, because you've left your job to go and train someone maybe You had to have someone to Shadow you or cover cover cover for you That cost is also part of the cost. So do the complete total cost of the training materials, manuals, meals Even research do detailed research for the training everything measure Identify your total cost for the training. Yeah So These are some of the ideas For the total cost for the training Then of course, what is number eight number eight? We're now getting into the calculation bit now. So number eight is Calculate the return on investment itself. You have enough information to calculate your return on investment Now the easy calculation or the easy formula to be able to do that is called the benefit cost ratio Right benefit cost ratio is simply benefits divided by costs So since you've already measured the costs What exactly are the benefits? So this is the formula benefit cost ratio bcr is benefits divided by costs So right at the time of instructional design Right at the time of writing your objectives, you need to understand what the benefits are how to measure benefits and how To measure costs you need to agree that upfront You don't come at the end of the training and then play with the figures You should agree that upfront Now it's also very important to keep it simple. So keep it simple And always involve managers involve managers throughout the process You need to involve them and really even educate them so that they understand the importance of what you are doing It's extremely critical that they understand that the importance of what you're doing And of course, you also have to give credit to participants And then in general all this means you're really planning for your return on investment, which is level four All right, so keep it simple So just giving you highlights of everything And this is key things I just said here. Keep it simple involve management educate management Give credit to participants and then technically plan plan your entire return on investment Right, let me do a quick demo. I don't like talking. Let me just open excel and show you what I did I did one tiny very simple demo But you can even do a more robust demo if you like, but let me see let me see if I can open that for you So here I have staff names. So these are staff Different staff staff numbers you can just say staff name and then maybe there are various tasks So, you know at that time when you are planning the whole process and you're looking at okay What is task one? How many hours were you doing it before? How many hours are you now doing it? So this is your before This task one it took me 30 hours to do and then task two Took me The same task one now it's taking me only eight hours to do So my total time savings is 22 hours. Let's say over a year or whatever Over your measurement period 22 hours. So this is the total hours saved Then of course you should have a cost per hour. So what is the cost to you per hour? Is it the salary? Or I mean Maybe he didn't do if he didn't do the task fast and somebody else couldn't do it So it's not as easy to as just a salary per hour. You'd have a general cost per hour Then you multiply that savings That's time savings by the cost per hour and they have your total return on investment So your total return on investment is 212,507 naira Then the next question is how much did the training cost? What's the cost? of Training right so if my cost of training is a hundred thousand This is just a simple simple maths cost of training is a hundred thousand that means At my actual return on investment calculation is I had a return of two One two divided by the cost So what was my return in percentage in percentage terms? So let me see So in percentage terms, I had a 212.51% return on investment interesting That's my ROI This is a very simplistic way of calculating ROI, but it's something that can be done The only way you can do this is if you plan ahead If clearly you said okay, what are these tasks that we were doing that we want to change by this training And then you measure it In detail put it down somewhere after maybe a month after the training measure again and see if there was any change But you need to isolate that change by having a maybe a control group. So yeah, it's not It's not that easy it takes a lot of money and many people don't want to spend that money But guess what technology is so cheap now that I think you can do it without spending too much money Right Okay, so let's quickly get back to our slides All right. So here we said keep it simple We did a quick demo for you on how you calculate return on investment is not that difficult So there were still 10, you know, there were 10 steps really So the next step which you have to calculating your return on investment is identify intangible benefits Because when you do that mathematical calculation, it's It's maths It's not just monetary benefits. There are many intangible benefits. There's increased job satisfaction. For example I mean people are happier because they're doing their work more efficiently. They are happy Yeah So more commitment to the organization by your staff for example increase innovation and stuff. There's so many Things that are intangible But then I I am of the belief that you can even calculate the intangibles Everything can be calculated, but I'm a technical guy. I'm an analyst, right? So everything can be calculated. So Yeah, so these are some of the ideas of measuring the intangibles so Then was the last bit the last bit for the jack phillip model of return on investment is You generate a study impact So you generate a study impact, which is you're actually going to sit down And do a more detailed analysis of how this affects your organization as a whole Yeah And then you now monitor progress in general and see How all your trainings must follow this process Of having a proper return on investment calculation engine Which really isn't too difficult to build really Um It needs a bit of even with excel is enough excel is quite good. It's just Getting clean data having your data clean is is very key So that's what return on investment is and that's the jack phillip model So I don't know if anybody has any questions Because I know I started 10 minutes later and it's already 3 10 So I just have one last poll for you guys If you have any questions, do you have any questions or before the poll? I know most of us here are experts learning and development experts But as the brand consulting we are analysts. We train financial analysts. We we analyst trainers really That's our specialty And What we do what I could do Or one thing I also do is I've written a book with about 15 other authors So 15 authors and I we wrote a book called destination facilitation. We actually wrote it with 80 80 d is the one that published the book and so the book is out and it's Going through how do you use your Or use your participants cultural Preferences or use your participants cultural traits. For example, you're training in a class that has a chinese and indian in nigeria and in south african Somebody from kuwait How do you bring all of that together in your training? And how do you how do you now take advantage of those cultural differences? So there's not really a minus is a big plus in the training So or for example an international trainer and you're flying to china Someone saw your online and liked your content and said hey come to china What are the things you shouldn't do things that you could do in the u.s That you should yes on how you become an Tata so his destination facilitation 15 of us wrote it 15 different countries covering the entire globe every continent in the globe And it's a really good read. Yeah, it's a really good read. I encourage you guys to to go check it out right I'll also share a link with you one of the key things in measuring training What I find especially for us that do technical training is a big frustration when it comes to Exams really Because at the end of the day when you're running an academy So you're running an academy And it's like a six months academy to me. I think six months is way too much to run an academy anyway So you're running this academy And at the end of the day you have to find out whether or not people How do I place people? Do I put this person in the copper finance department? Do I put him in the treasury department? And what do you end up doing is writing an exam? So people now write this exam and The exam at the end of the day is what you now use to place people But there are some people that are specialist in writing exams I mean I've come to realize that there's some people that are just so so good at writing exams So once they read and do the exam you now place them based on that exam Is really not a good way a better way is throughout that training process. You're doing level two You're doing level two level two level two level two which is gauging whether or not The skill knowledge and attitude has actually passed during that training during the academy So you remember the clicker thing I said we use you can use that every single day Gather data points on whether their competencies whether or not They exhibit those competencies every single day So you're testing different competencies for the different things you're training Use simple quizzes for that at the end of the day if you if you measure 20 20 data points every day and you're doing something for three months less even assumes 100 days 20 times 100 is 2000 data points per person Those 2000 data points will tell you a lot Now if you map those 2000 data points you automatically know exactly where someone's strengths are You don't even need an exam at the end of the day So there's another way you could also do this to me One of the core skills for an analyst is financial modeling And I'm so happy that there's finally a certification worldwide certification That measures precisely whether people have the skill And knowledge for financial modeling the attitude is a different thing But the skill and knowledge is measured during the exam and the way they do it is this the exam itself In the exam you have a laptop You do you don't have a piece of you have a piece of paper But you really have a laptop and you're meant to build a model from scratch So if you say a financial modeler, well, can you build a model show us during the exam? So in the exam you build a model a model from scratch Now after you build a model you answer your questions You're building the model and answering questions a full four hour exam And once you're done after the exam. Yes, you know how to build a model because you built it at the exam proctored exam And glad to say that they are now finally going to have a center in Nigeria And these exams are going to happen on the 28th of april. So whoever In your team needs this financial modeling Please let us a very detailed course that we're going to do in march that covers the exam And it gets people ready and we have an online course as well. So we have online trainings That we offer so online trainings if I go to office training hub, you'll see our online training So we have lots of online platform where we do various trainings So this is our online platform. We have training for every kind of analyst and we Constantly loading new courses on there and we're going to load an advance for your school So these are measurements of evaluations and stuff if you change even the way people train and the way people gather the knowledge And put things like this exam for example fmi and during the exam you're actually showing whether you have skill Well, we'll measure our attitude but skill and knowledge directly you either pass or fail based on skill and knowledge So thanks everybody. I hope you enjoyed this webinar we do this every month and Join the next month if you want me to send you an email to join or if you have already Had we recall you could just tell me on the chat say yes. Yes. You just type yes on the chat if you want me on the next one And we will look for you at the next one, but Before you go one last Last quiz, uh, let me pull that up So I have a poll So all I want to know is what's the highest level of Highest level you've ever measured So what is the highest level of training evaluation that you typically perform in your organization? Level one level two level three or level four So this is just the last question for a close Ah, are you sure someone's saying level four return on investments? Who is that says that's that's amazing Yeah, okay Return on investment you have actually measured your return on investment. So that's cool Well, level two sounds more like it level one So the highest level you do is level four. That's good And then level two so most people do level two most people I know it's level two and even level two they don't do it as well as as I I think they should to measure skill attitude and skill knowledge and attitude and then then two new Levels of level two the two new line items, which is confidence and commitment Yeah, okay guys. So thank you very much and we hope to see you in the next webinar Next month, which is the third Thursday of every month. So we see you next month guys. Thank you very much for joining us Next month. Hopefully our speaker will be able to be available and we have the speaker. So thank you very much and bye. Bye everybody