 Before we start, so this is going to be very interactive and we're going to do some thinking and talking and working together, hopefully. But before we start, I thought it would be good to just go around the room very quickly and just for each person to say, what is their name, and then answer one question, and that question is Jim Rulmore, Josh Jarrett. We'll start right here. Charlie and Jarrett. Just maybe stand up and tell everyone, kind of show everyone who you are. We can do this quickly. First of all, I'd like to talk to Jarrett. Tomo, I'd like to say Josh. David Peterson, and I'd like to choose. Carly Sillian, I'd like to get rid of that whose name is Jay. Pino Rahman, suddenly I picked Jim Rulmore. Denise Wender, Jarrett. James and Josh, definitely. Bob Gabbins, and Jim. Emily Seiber, undecided. Charles Snare, undecided. And Jonas, both. Oh, is that Trent? Yeah. Paul Lertz, Groom, we go to the St. Barber. I'm sorry, I'm sorry. I'm sorry, I'm sorry. Mike Mather, Jim, Groom. Jay Heate, Josh. Andrew Maleficent, Jim. Michael Valstein, I want to see him a lot of child. Sarah Loose, definitely the one in each world. I'm just groomed. Mike Silver, I like Josh's grooming. Preston Parker, too. Heather Leary, in the study. I'm also groomed. Quickly the last question. I can't hear, I don't know about it. We're going to read this about them. Not about that, I'm friends with Joseph. I can't read about it. John Mayer, it's not about who you like. I don't have any experience. I'm wearing a sports coat, but I still vote. I'm going to ask you to Jim Groom. Jane Gallagher, vote. Scott Ashton, Jim. Kevin Ashton, vote. Papa Cumbering, Jim. I don't know what we're voting on. You're actually not voting, it's just a question. Jim Groom or Josh Jarrett? Oh, Jim. Someone in the door there? I don't have a question. Jim Groom or... Oh, um... Yeah, there's some of me who can be married to him. Carl Schiffler, vote. Chris Doppler, vote. I'm sure he's not challenged. I'm sure he's not challenged. I still vote. I'm going to Jim Groom. What was your different question? Eric Hopper, vote. Luna Daly, vote. Robert Hart, Jim. Morati, much tomorrow. I like both. Then Wilson and Josh. Ira Gooding, the head says Josh. The heart says Jim. Ariel Diaz, Josh. Jeff Davidson, channel foundation. Yes, to vote. Kim Emery, vote. Sarah Carter, Jim. Royce Kimmins, Jim. Satoshi Amalaki, this is my proposition. Okay. Colin Clark, I slept in this morning. I slept in this dimple in the fall. I slept in this deep end group in the fall. I was looking at Demi and Jim. Okay, Jim. Sean Hurtman, definitely vote. Harash Bissell, Jim Jarrett. We have more seats here at the front, actually, or on the sides and kind of... you guys want to sit? You can stroll. Leave in the middle. I'll be right about it. I'll be right about it. Okay, so I think that the set of keynotes is actually going to shake up what we can talk about here a little bit in a good way. But I'll just quickly run through some slides that I put together to provide a little bit... as little context as necessary, I thought. And I'm actually also a little surprised that there are so many people here because I submitted this session idea kind of a little bit tiny and cheek, and I wasn't... first I didn't think I was going to make it into the program. And then I actually didn't think that there would be over, well, however many people are here, with many people. So, can you play is kind of that the moment when, as a kid, you're coming into the playground to play football in Germany where I grew up, and you don't know the other guys and then there's that question, can you play? And I'm sure it's the same in the States when you play basketball or any other sport. And it's that weird moment where they don't know anything about you, maybe watch them a little bit, but then you're nervous and how well can you play. And if you could... In the States you say, do you have game? Do you have game? Okay. Do you have game? So, and you kind of... you know, there isn't really a way to show them a passport or like, yes, I graduated from the School of Football in 7th grade, and so we're facing these situations in our lives all the time where we have no certification for things that we can do or who we are and maybe that's good or maybe that's not. But why did I think that was a useful topic to talk about at the Open Education Conference? The reason is that it seems to me one of the big pushes that this community is kind of being confronted with or one of the big areas that people are saying we should be moving into around assessment, certification and credentials. And there are lots of interesting things happening that we could kind of dive into including some of the work around badges that a lot of people here are familiar with and Carla who's sitting in the front here is actually the project manager for the Open Badges Infrastructure Project at Mozilla. She works with Erin Knight, that's someone you might know. Who just had a baby on Wednesday? Who just had a baby? She immediately received a baby... What is it? The baby badge? I will pull that up later and show you guys there is that badge. There's this focus that's shifting certainly into analytics and how can we track what people learn and what can we say about their competencies and it's become a big focus area for Open Education. There are these statements that we need some validation, we need standardization so we can compare success we need to be able to have kind of compatibility across different sets of standards and at the beginning of a very interesting trajectory where we're going with this so I think it's a good time to talk about this kind of in a more general way. And then the kind of hesitation I bring to this is some of the stuff I'm observing and hearing and so there's for example James, I don't know if it's G or G but he's a kind of well-known researcher around games and learning and digital literacies and he basically said testing is primitive so any kind of testing that you do to assess learning is primitive and will never produce any really meaningful results and so he's come up with these pretty strong statements against testing and then people are arguing around that it's like games don't have tests if you work through a game you kind of get to the next level when you've achieved a certain level of competency but no one says stop, let's do a test if you've really achieved those things but you actually, you know you get onto that basketball court you hit the first three point attempts, it's like it's clear that you can play and they don't stop you there and say how many meters is the three point line away from the basket how high is the basket who shot the most three points in the 1979 season it's like it's part of what's important about playing basketball which is playing basketball and communities likewise we don't really have tests and that is also intellectual communities where we have very sophisticated patterns of new people coming into the community often through processes of mentorship or apprenticeship and it's focused around the activities that the community cares about and in that process assessments are made feedback is given, learning happens but there rarely is kind of a test that is the actual projects that people are working on the learning that happens and so there's a certain feeling that these games and these communities they know people's competencies and I think in that tension between the desire to test and quantify everything and to just know everything and there are problems with this as well I think, somewhere in the middle there is an interesting space and I'm just going to give one example as a story we need to include stories in presentations that's the only way people listen and remember everything so there's Auguste Volna who's arguably the greatest the father of modern sculpture and one of the greatest sculptors in the history of sculpture he, when you when you look at his work and his life there's some interesting things about assessment and certification that we can take from that and one is I think about the process of assessment so I thought about this a little bit there are always constraints that are part of assessment you always have to focus either you need a series of questions or you have a series of multiple choice options or you have a certain time limit to produce something but assessment always there are certain constraints in a very broad sense and in his work the constraints of assessments are really set by the stone and the tools and then his ability or his competence is the process of using kind of pushing up against these constraints and creating a work of art that happens in that tension between the constraints and his creativity and maybe that so I don't usually talk in that anyway I feel like I'm a thin ice here very artsy farty things but I'm going with it anyway so I think there's something about constraints that's kind of an interesting concept when we think about assessment and what are good constraints and what are bad constraints and what are meaningful constraints and how do we make sure that the constraints are meaningful and actually useful and then there's something about measurements so in his case he was never admitted into the great art school of France that was his dream was to go to this great school and he applied three times and he was rejected three times and at the time the requirements to enter the school were actually fairly low it wasn't so hard to get into this school and so for him to be refused three times was a pretty crushing defeat and thankfully he carried on he became more of a craftsman for a while so he worked his way up and eventually got to the point where he had the freedom to pursue his vision but I think what this says about measurements is that we're creating certain measurements or standards or qualifications that we expected the history has shown us over and over again that often the people who are doing the measuring are the wrong people and the things they are measuring are the wrong things to measure and so I just want us to keep that in mind as we think about assessments that if we look at the history we basically and this isn't just in arts it would be easy to discount this but what about Galileo the things we were looking for in the history of science often if we had measured them and standardized them we would just listen to that how do we make that next step to the things that we don't know yet we should be looking for what could this world of assessment and certification look like if we what could the ideal world of assessment and certification look like I think we can draw some inspiration from games we can draw some inspiration from sculpture the constraint of the material and the tools we can take some inspiration maybe from chess because it's so implicit in the act of playing chess that I never need to stop and ask you if you play chess against me you're most likely to win against me by the way and then I have some assessment of your ability to play chess and you have some lower assessment of my ability to play chess so my question that I would like to discuss as a group is kind of around how can open learning and really also pushing on the idea of open why is open different than not open learning how can it be authentic or engaging or social so that it produces all the necessary evidence of achievements as a byproduct of the learning so we don't need to stop so this is kind of my broader question and I think I I was going to do this next but I think I want to pause let me tell you what I want to do next but then I want to pause and I want us to spend a little bit of time discussing the keynotes because I think there's sort of rich stuff from the keynotes that actually is directly related to these questions but so what I want to do after the kind of open discussions I want you to form groups of two people so just grab the person that sits next to you if you know the person that sits next to you probably grab someone else someone you haven't spoken to and spend 5 minutes or 7 minutes thinking about assessment and certification in the open education space and try to find one example or try to come up with one story of the perfect kind of assessment in an area that you care about so if you work in a certain content area and you think about assessment how it's done and you think about all the dreams you may have for assessment in that area what's a perfect example for amazing assessment in your area and then also one example for an area or a case where the kind of more inspirational some of these examples I've talked about where that will never work where we need the standardized tests where we need the multiple choice reviews where we need rigor and standards structure and we need to have committees that agree on those the institutional level of assessment where we absolutely need that so those are the two things but let's pause and by the way the etherpad here everyone can take notes in there you don't need to log in or anything you just open it you can see other multiple types don't use the chat there's a ping attached to the chat so if you start using the chat there will be endless ping in this room coming out of all your laptops but if you want to start taking notes or putting stuff in there there's like a shared note taking space available let's not do this right now I thought what would be most useful is especially for the people who have been to the keynotes um um is to maybe just kind of general reactions on it seemed to me like they were talking about two extreme ends of an open education future and on both of those ends there are interesting questions around certification and assessment and competencies so I'd love to just maybe open the floor to anyone who has reactions or questions or kind of when they were sitting and they went oh I really want to ask or I really want to say something we didn't have time at the end which was frustrating so I thought maybe we can do some of that now so I'll open the floor there we go I think both of them point into one of the things they both point into was that the system of education as it's been over the past centuries is breaking down that's working if it did work that's the other good question did you see any implications for the testing and assessment and those kind of things I have a lot of thoughts about that myself but let me just put it out in my opinion educational institutions do two different things and they can really be opposed to each other and one of them is teaching and the other is providing connections and one for some people the ability to learn is really hindered by the fact that somebody is sitting injection of what they're trying to learn this I've been really struck by your older Ivan Illich de-schooling was a huge thing what 30 years ago whenever it was and everybody got excited about it and credentialing is probably 10 times more powerful now than it was then so why, what happened because it was inspiring and it sounds like it's all happening again with the same kind of question and yet credentialing is the one thing that got sort of reinforced yeah, I guess and credentialing is for me is only one part of the problem the way I've heard those two talks is the the tension between empowerment and skill and I always go back I thought that Josh's story about Briana was incredibly moving for me what it's always about so on the one hand there are people like Briana out there many many many people like Briana out there who need help getting living lives of dignity and economic self sufficiency and so on and there's a lot of sort of objective credentialing kind of stuff you need to do and credentials are just one of the devices the very very imperfect devices that we create to try and scale that on the other hand I think when you listen to someone like Jim or Gardner what I get from them is that before they can achieve their goals in terms of learning these skills and so on they have to be empowered and talking about empowerment at scale and focusing more on because empowerment is fundamentally personal that's what I got from Jim's talk more than anything else people brought to that course what they have and what was important to them and it was the core inside of that course was I can bring that and get value out of it before I can learn to do anything I can learn I can do something and that's what that course teaches so so that's what I saw as the tension and it reached it I actually thought despite the differences there were a lot of common messages in both talks and one was exactly that about empowering students and Josh was talking about these assessments so that students knew at the onset of a course what their probability was of actually completing and what that information did for the students so there's empowerment and you could do that kind of empowerment at scale and make it more human centered really and more about the experience the other thing about I mean obviously Jim was talking about experience and I'm really excited to hear that conversation at this conference but Josh was also talking about it it's not just about resources floating out there it's how are you going to create that system where they're used really effectively and combine with all the things that we know about learning so I'm not going to give up on scaling empowerment and personalized learning and experiences of learning I think that's so important I'm still thinking this through but I think that one thing that both of them at our talk now is the need for multiple assessments and I think the reason I've resisted to choosing one over the other is that I'm nervous about trying to find the perfect assessment model because I think that and I'm still thinking this through different assessments might work best for different people I'm very nervous about standardized tests but I also know some people who weren't so great in the classroom because classrooms were boring but were able to show their knowledge and intelligence on standardized tests I know other people who don't do well on standardized tests that do great in the classroom or on particular assignments so I think that what's important to me is finding the multiplicity from the different talks for different sorts of things it strikes me that this was a lot of empowering students going to the public schools and you say are the teachers empowered I don't think so this is certain and why not partially because of all of the strengths of my stuff so they don't feel like well what do I feel is best I can't think about that that's nice so I think before I can really engage in the conversation I ask the question accreditation or assessment for what is it to get a job is it because it's a scorecard at some point in our lives to say we've engaged in a social activity to become better informed citizens there are a lot of folks that are very concerned in universities and the major part of what college universities do is to help create better folks that are better educated and that forms a better citizenry which makes a stronger social fabric and it's not necessarily about getting a job and there's tensions there I think that gets very interesting when you talk about badges because clearly if you're an employer and you've got a specific set of skills and competencies that you want and if you're a student looking for a job that's really your goal certainly in today's day and age that's probably a big goal of a lot of students then that's important so I'm with you you're asking what assessments might be best and I've got a lot of questions about why are we assessing and for what aims and I think that there's a lot there's different pathways there but we need to be transparent about that so when students come in to an educational environment whatever it might be transparency about here's a pathway of assessment it's really about getting a job this is a pathway of assessment it's really about the liberal arts education there's different answers liberal arts education is never getting a job hate certainly can't hold alright so I come from a liberal arts school and a very good one at that but listening to Briana's story I think that with what Jim Sheldon and Martha had to say yesterday I mean it's very obvious that the current system has failed people like Briana and so what I would like to us actually or someone to take the leadership is to go back to people who make the policy on accreditation because they hold the gates for new models to evolve so when you have new models of education which are looking at other ways of educating people like Briana they shouldn't be held back because the sex accreditation isn't available or whatever you know whatever the thing is so I think it's in conversations like this and in meetings like this we should be able to empower people who are willing to put out new models of colleges that are completely unlike where I come from which is a wonderful college but nevertheless we will never be able to serve the Brianas of the world and Briana's of the world shouldn't have to hit our doors and fail because there is something about them that's good and valuable we just need a different system a different college model or an educational model to give them the ability to say I'm educated and I am employable and go for them so it needs to go back to the Jims and the Martha's three people maybe Carla is this a direct look like what sort of badge is but I think there's lots of great questions the one thing I think about when I think about badges is learning is life long and we're really talking about kind of a subset here and they're all kind of things that people have skills and competencies in they don't get recognized for but they make them really great citizens or really great partners or really great workers and so I think there's lots of opportunities so the big question is like okay so what kind of assessment are we talking about and then who are the right people to be assessing the SSU does it have to be top down but then likewise also addressing what people was talking about you know from a governmental standpoint what are the standards that kind of exist right now and can we start to think about them in a brand new and broader way I'm the executive director of what's being when you say that current systems have failed I don't understand what system we're talking about we're talking about our society we kind of often say it's one of these people okay who talks to us and we kind of community colleges I'm talking about society I think that what I would be preparing is going to be right what I would be preparing for and we can start re-envisioning current systems but we can also start envisioning new systems to serve people in what we want to be doing I just thought it was just that I was just thinking that a test or an assessment isn't necessarily a bad thing it can be an illuminating thing it's just that we've often attached a certain stigma to kids who or even adults who fear failure I mean a failed test is not so much a bad thing as a great opportunity because now you know what you don't know before you take the test you're sort of you probably think you know everything I mean you wouldn't walk into a test if you didn't think you knew what the material so a test is not necessarily a bad thing it's just that you failed them and judged them and I think it's a great opportunity oh great this is awesome that you failed this test because now we can help you on these six questions that you clearly need a little bit of help with so I guess reframing it can have a positive impact I think video games are good examples or chess too if I lose to you in chess and I never play chess again and just go on to basketball that's not helping me learn it's helping me forget all the opportunity I could have had for you to teach me or someone else about it I think this is an excellent point I want to just respond quickly there's been some research recently about learning the importance of failure and problems and the research showed that the key thing is how you deal with that failure so if you we all fail right but if you fail and the failure happens in a way that encourages you to then kind of try again and find the answers that's how learning happens if you fail and then you go play basketball then no learning happens essentially and apparently there are these two broad strategies that we develop for dealing with failure and they're related to encouragement of expertise and motivation so if we tell our kids that they're very smart when they fail it's a big shock because all of a sudden they're catching their identities, I'm smart I'm failing, it threatens my identity they then tend to not try again because they don't want to have that bad experience if we encourage them to work hard and we say it's amazing you've worked really hard to get this they start developing a sense of you get somewhere to try and when they run into problems they'll try again and they end up learning more which I thought was really interesting I think it's related to how we assess them what are the effects of the assessments do we encourage you to try again or do we encourage you to go somewhere else I put a leader of times article about that perfect so there's a hand there and then we have one there it occurs to me that when we're talking about you play basketball and you lose and you never play basketball again we're talking about finality versus something that happens in an iterative cycle you play chess and you play another game you play another game you're iterating through these things I think that's part of the key if the assessment comes in the end then you're too late you just have a question I don't know what I don't know but we're moving on so I can't stop and go back but if it's iteratively working that's a different kind of story and the other thing that I was thinking I'm not sure where this part gets in but when we're talking about things like badges and proof it strikes me that we're really close to the idea of a portfolio of work but that's the whole website well that's the thing I go look at the website I built if it's a massive because it's horrible then obviously I don't but that's the kind of most honest about there is, this is what I've done I don't remember, I think you were next and then here's a hand so I think one of the things we tend to gloss over when we talk about authentic assessments we worry a lot about false negatives that we're not giving credit for people who deserve credit but I think we also need to worry about false positives I think we need to spare our thought for the coordinated souls who graduate from Ivy League schools thinking that they have necessarily learned something because they've graduated from Ivy League schools we actually have a pretty pure test of that here which is the financial services industry if anyone has ever talked all those folks are directly out of tier and top tier schools if you've ever talked to a sales side stock analyst about what they think about educational technology companies and the root dominant and why you have pretty good evidence that these folks have not been taught the critical thinking scales and they come from good elementary schools good high schools all that not their fault just what the other gentleman was saying our educational system is geared towards a prior age and so it's become this engine that has a life on its own so people learn to be good at school and we don't know whether that's what they come out with is of any use at all they can't tell you why the seasons happen either they what? the Harvard grad studies show that the graduates couldn't tell you why the seasons happen either whether it's because of the surroundings which is that's great and I think it ties back to the idea of any kind of credential is a proxy it holds meaning and people attribute different types of meaning to it and that credential for an Ivy League college we all have ideas of what that means and those ideas are grounded in a long history ideally or there's some evidence but they take on a value of their own no way to actually now go back and look at that credential and then drill deeper and say what did this person do exactly do they know critical thinking we bundle it all up into this very very high level aggregate and we hope that important things are in there I think there's a head here then so the point I was going to make was just what you said which is thank you proxy it's very imperfect blunt instrument and there are different degrees or details getting a degree from certain school is one level but getting a grade for one school for one course is another and you can go down to minute details if he did answer this question that way or that this way he wrote this essay for this assignment and we cannot really deal with infinite amount of details to get the society to work or economy so we rely on those proxies for the sake of efficiency but inevitably fails it does injustice to individual cases fail to sometimes capture the whole massive things because of the misconception of what it represents what it doesn't but at the same time I don't think we can just all together throw it away and I also have this idea that you know it is a lifelong learning kind of thing and we need to have that kind of proxy in order for how do you say to promote the circulation of information about who knows what who has that what kind of skills and if we don't then we might end up with even more imperfect signals signals in the economic sense so he did that or he's a friend of this he is trusted by this guy I might be the only signals that we might rely on which isn't that good as compared to some more specific badges it's my guess I would like to raise a fundamental question being from the Netherlands you might expect that maybe how assessments are far from perfect we all know that and I'll give the example of a student at the University of the Netherlands passing a philosophy exam it was a multiple choice exam 80 questions and he did not pass the exam and after asking the examiner what would happen if he would have an oral examination and the examiner said well he would have passed at a very high level because I knew the students was very good what it shows is that the way you are examining students is not as far from perfect and because that's true and also because we are referring to because of the limitations of our assessments and mechanisms I doubt that you are observing now that you can uncouple the learning process and the assessment process but that's my fundamental question it's in the heart of this discussion I think I'm much more sure about the added value of the things the example of philosophy students shows that because the examiner knows the student from the learning process he knows that he should have passed the exam and I'm sure we can improve our assessment very much but I'm sure we cannot improve to the perfect level so what does that mean? I'll start with coming back to the intro question around Josh versus Jim Josh lays out a really analytical framework for how to think about all the challenges and problems and Jim provides a very inspiring example of one way to really achieve that and I think for that reason they actually work in cohort and one model of education throughout there and what that leads to in terms of lifelong learning in terms of improving everything it's really about inspiring curiosity and providing tools and tools evolve across the educational landscape and providing curiosity something that we do our best to feed our children at a very young age but if you continue to inspire that curiosity and continue to provide the tools as those tools evolve and evaluate those tools and if you discretize a lot of the core tools you can actually get to a very deep understanding of what's the foundational tools around logic, around reading comprehension and then you can continue to build off of those and then once you combine those and how you really assess and teach which should be combined and the question is how you scale the assessment if it's an integral part of learning which becomes a more difficult challenge to scale which is part of the broader question but I think that's one framework I think Josh put up this analytical framework Jim has this inspirational example and how do we synthesize this curiosity and measuring tools and not to evaluate and promote failure as somebody mentioned but as a way to know where are the tools deficient and how do we improve those tools and then how do we continue to drive those tools to achieve broad things across citizenship, across career paths etc. So there's one more hand there and then I'd like to shift gears What's been missing for me in the discussion is a discussion around the learning targets or the outcomes and the question of validity and I'm using the assessment side Traditionally assessments are not the same thing and when you design assessments you have to understand what you're assessing what your targets are the goals, the context and the various forms of evidence that you have to collect prove or try to prove that outcome or that target would be met but it doesn't really matter what the results of the badge or anything else and then the validity of that if it results into some sort of credential find is really about to lead those assessments and how that was designed what I didn't see and hear about from Josh's discussion was anything about what the students were supposed to be learning the rate, destructivist, experiential type of environment totally agree with that but I didn't know what they were supposed to come out with and the word point knew all of this that we were doing that in Jim's example in Jim's, I'm sorry thank you and Josh did mention some things about a lot of the targets but I was in a different discussion but that wasn't so when we think about this that's what's missing for me we've been in adges and you start discussing let's talk about validity of attainment evidence of attainment that you've met this and how you're getting there and then when you say yes if you play chess and maybe the alchemist did you wing a chess in some cases you may not care on how you won okay well that's an outcome and that's an assessment and actually in chess they have very sophisticated measurements then that calculate my score based on what I've beaten so I would love there are two more hands no but I would love for us to actually move into the practical because it's always I think it's nice to stay in the theoretical and kind of engage with the ideas and then to walk out the door and kind of have an interesting conversation but it beats the next step is really to get our hands into what this could look like so I would love for us to try to think about concrete examples and the two kind of ideas I had is if you could just grab the person that sits next to you if you know them or you work with them a lot grab someone else and spend five minutes to really thinking about maybe paint one of the two either an area where some of the more out there ideas no more credentials no more tests where that will never work and for good reasons people are running out but now which is fine or pick an example where that is close to your heart where you think there could be a much much better way of doing assessment and credentialing or assessment or credentialing and you map out maybe in a few bullet points what that could look like who is the source of validity what's the mechanism of doing the assessment what is the assessment done for so that we can connect a few kind of examples or case studies which will then help us in the next step to think through all the complications so I'm ready set go I'm going to check I'm going to check about five minutes and then we have at least a little time for this is a very unruly class which is a good sign so there's a lot of juice in this which is great but I wonder so two things one is if you have any chance to put some notes into this etherpad of what you just talked about with your partner please do because I would love to see some of that stuff and then secondly can we just have one or two people who want to maybe give a little rundown of what they talked about so we've got one volunteer two volunteers maybe a third and four you only have about one minute each and we're already running over so we'll do it quickly we talked about portfolio assessment and how portfolio is a demonstration of what you know rather than credential the example that's difficult to measure is doctors where you think one way to measure whether they're good or not is malpractices but that actually correlates to how they did it more than what school they went to it's all about the personality in terms of not being sued so that's an area where almost any assessment you can do is orthogonal to what you're actually measuring and then an area where it's actually positive is things that you should be able to algorithmically assess and scaleably assess on a progression scale which is why video game work because you don't just randomly buy a huge character you buy a little character and then bigger characters and bigger characters like the three order production so you learn that scale which is something that's very, very measurable in math and reading for example Excellent Andrew, we talked about Jay's experience mostly in vocational instruction and how obviously the best way to know if someone gets it is to walk inside the brain if you were a psychic and that would be ideal since we're not we'd have to use other sort of proxies and things and generally speaking we were saying the best way is just to observe them doing it whether it's doing a balance sheet or rebuilding a transmission or doing woodwork or I suppose even mathematics if they're a way for direct observation and I suppose even I'm adding this coaching after the fact to just talk about, talk through the problems I think that would be your constructive means of assessment and anything else So we're just building on that so we're going to contrast two things twice Firstly, there's an important distinction between knowing certain specific things or just sort of content or actual knowledge and then there'll be the other thing which is knowing how to do something or just in general how you approach solving a particular problem that's really about, let's say, using a standard line for the global context So that contrast and then building on that, I think it's important that the advantages and disadvantages are two of these two ways of testing The one and the sort of the big hammer where you try and test just at a base level what people know so base standardized testing and you really have to know that that is a minimum kind of test that doesn't really tell you much about a person but it tells you that it gives you sort of a lower bar on what they know and then on the other hand it's not the base but more sort of the simple example I've said on T is just assessing really how so the things that are hard to say is like how people approach problems I think it's done much better through sort of almost like a network of trust kind of model where if somebody else from UTrust tells you that this other person is really good at doing tests then you do have good reasons to be trying to trust this person I think that's much more expensive but a much better way of assessing going into practice here Can we have one more hand? We talked about an area where I think in general our kind of discussion is about how there are different kinds of assessments we talked about how like a brain surgeon you would want to know that they knew their biology you know that's something where you kind of need to make sure that they know the parts of the brain it's really important something else I was thinking about is like that assessment doesn't have to be black and white, master or not I think always I think sometimes it could be okay people know this in this context or here are things that help people succeed that it's not necessarily always going to be that brain surgeon like they need to know the parts of the brain but hey this person counsels really well when they're working with this type of person somebody who's outgoing say but this type of person somebody who needs to be drawn out more they're not a good counselor for that situation or they need these structures to be a good counselor in that situation and that it's not sometimes black and white need to know the brain but that it's not always that simple okay well actually Kyle was going to raise his hand but I'm going to ask you not to because I will what we didn't talk about very much is the badges infrastructure and I will come back to that a little bit in my presentation tomorrow and so I think that's probably okay well anyway thank you all very very much this was awesome