 I hear when you're standing there I do, yes. I'll start the intro then. Hang on, let me just get back over to you. There we are. Okay. Comments, welcome to our final session at ALT-C 2015. So I'm delighted that our keynote speaker is Phil Long. Phil has been associated with a host of learning technology initiatives, including advising the new media consortium. And if you've seen his biography within the program, you'll see how far his influence extends. And Phil was previously director of the Centre for Educational Innovation and Technology at the University of Queensland, and he's currently Associate Vice Provost at the University of Texas, Austin. From where he is joining us today, Phil, we are looking forward immensely to hearing your thoughts about the future of learning technology. Over to you. Okay, thank you very much everyone. Hopefully you're hearing my audio reasonably well. And what I'd like to do is switch over to the screenshot of my desktop, which should give you a gorgeous view. The screen just, my camera just got moved around, didn't it? A gorgeous view of the night sky. And from there, I hope that you'll see my first slide. Is that affirmative? Yep, we're good. Okay, so I'm going to talk a little bit today. I'm going to try to keep this short so there's opportunities for question and it's the end of a long conference. But first of all, I wanted to thank you all for inviting me to speak at ALT-C. And after hearing and seeing the presentations by Laura and Rebecca and Jonathan and Steve, I have to say I'm a bit concerned to be the cleanup player of this particular conference, but I will do my best. And a special thanks to Amanda, who was a delightful presence at the University of Queensland when I was down there and she was able to join us for a while and found the presentations and work that she was doing with us was terrific. So thank you, Amanda. Let me start by just giving you a brief narrative arc of what I'm going to talk about. I'm going to talk a little bit about performance versus learning. The theme of this talk, as you saw from the original slide, is in fact, why are we not using what we really know about cognitive science and learning as well as we should, both in terms of the way we design presentations and engagements and classes, as well as the way in which we design and build our tools for facilitating learning. So the narrative arc of this talk is to talk a little bit about the distinction between performance versus learning, which is, I think, a major problem that we have today. Secondly, a few examples of what we really do know from over 100 years' worth of cognitive science and learning research is effective and works and some of the issues that it presents. I'll segue briefly to a bit on metacognition and the self-awareness or the learner's awareness of their own abilities, which is, let's say, the depressing part of the talk, and then move to a little bit more optimism, I think, around transcendent motivation, which I think is going to help us as we move forward. And then I'll conclude with a few questions for you to think about. So the next step, let me ask this initial question and the initial question is what I started with, is why aren't we building the digital tools for learning environments with knowledge of and incorporation of what we know about learning science and, of course, why are our course designers in classes equally not taking advantage of that understanding? And there's a good reason for that, I think. But first, let me talk about what I mean about learning. And so in this case, I'll grab from the Oxford English Dictionary the definition of learning as the acquisition of knowledge or skills through experience or study, as in being taught. And then a little bit more clearly, I think, from a more recent article this last year, this current year, by Sorterström and Burke of York, learning reflects the relatively permanent changes in behavior or knowledge that support long-term retention and transfer. And therein lies the rub, long-term retention and transfer. So we know what we seek about learners in terms of learning, at least most of us who teach have an idea of what we're trying to accomplish. In this particular case, again from Sorterström and Burke, students can use information in available context, use the cues available to reconstruct knowledge to meet the demands of a present activity. Now that's distinct from performance, which are temporary fluctuations in behavior or knowledge that can be observed and measured during or immediately after that initial acquisition process. And that is a key distinction. Now learning involves multiple types. Of course, things like latent learning, latent learning consists of learning that you actually do without really being aware of it if you happen to be walking through a park, for example, waiting for somebody. The reality is that you are picking up attributes of the park, the people there, the location of objects in the park, and things like that. And if someone asked you later, you would be able to tell them something about that, even though there was no overt reinforcement for the idea of learning anything while you were in that park. On the other hand, overlearning is when you present someone with a series of act tasks and they practice those tasks, they get rewarded for those tasks, and they begin to demonstrate competency in that they can reliably reproduce or act on or use that knowledge in an effective, consistent way. Oftentimes, that's where we and our students stop. Overlearning is the continuation of that process well beyond that demonstration of initial competence and mastery. A classic experiment in this had to do with overlearning a series of words in a list, a classic psychologist ploy, and per doing that with one group where they had to learn the list and recall all the words. In another group, there was a control group who did the same thing. The difference between the control and the treatment is that the control group stopped their learning process when they demonstrated 100% recall in that mastery of that. The overlearning group continued that practice and did so for another 27 days. Then they then came back and did a retention test and the people in the overlearning group were much, much more capable of remembering those activities. Indeed, if you did the same thing and you just let an hour pass, just any period of time, the people in the overlearning group significantly improved their performance relative to the people who stopped after the initial mastery. And finally, fatigue learning I don't have to talk about because that's what we usually do every day of the week since we're usually fatigued when we're learning in our typical work days and our school site environments. But you know what I'm talking about. You're learning even though you're tired and the like, we tend to think we're not in fact picking things up as well as we otherwise would, we're actually doing better than we think. And fatigue learning does in fact result in reasonable retention. Now, the thing is that with learning, whether it's latent learning, fatigue learning, whatever type of learning that we're talking about, we can have learning taking place as in latent learning with no discernible changes in performance. And this is to say when I'm walking through that park, I'm not changing my behavior because of the things I'm picking up, but if you ask me later about attributes of what I saw, I will be able to tell you. If the converse is also true, that is I can demonstrate improved performance and yet fail to significantly learn in the terms of long-term retention anything. And we all are experienced with that because all of us have administered and or taken midterms and finals and the week there or two weeks thereafter the retention of that material is often poor if non-existent. And finally, we can discern sort of the most interesting thing that there are certain manipulations that have opposite effects on learning, things that really, really make performance effective, but learning non-existent and things that really affect learning, but performance isn't so good. And the latter, when things affect learning but performance isn't what we would like, is in fact one of the major issues that we have to confront. Now, let me talk about a couple of the things we know extremely well and you've all probably been well-schooled in this, but I just want to mention that things like space practice. Sir Rohr and Pashler did a very simple study on space practice using a very common paradigm. You study a material for a period of time. You leave a gap. You re-study the material. You have another gap and then you have a test that you present to understand whether or not they were able to reconstruct what you were trying to teach them. So, the thing that's key about this is that it's not a linear relationship. So, Sepeda and others in 2008 looked at 1,354 study participants in a study and they presented them with a range of time frames, study gaps that range from 20 minutes to 105 days. And followed that by tests for some as short as 7 days after and to others as long as 350 days after and the various spots in between. The optimal study gap that's listed there is shown by that red curved line on the saddle. And the point of this slide is really to convey that there is in fact a nonlinear relationship between study gap and test delay. So that, as you see in the saddle, it looks like it's between 5 and 10% of the test delay is the optimum gap in that study for this particular experiment. So we have to be mindful that memory retrieval and performance and long-term activities, long-term learning, is not a simple linear function. In addition, we have things that we know very well about variable or interleave practice. A couple of classic experiments, one of which is really very relevant to the higher education environment in which there were students that were presented with opportunity to study for an exam and they were given the opportunity to study for the exam twice. In one case, they studied in the same room each time. In the second case, they studied twice, but in the second room for the treatment group, they studied in a different location, a classroom, but a different classroom as the image is sort of depict. And the test was taking place finally in a third space altogether. And the students that studied the same material the same length of time, same study gaps, et cetera, who changed rooms between in the second study interval performed significantly better than those that stayed in the same environment. Again, an example of interleaved practice. Another example which is from the studying of art where in an art class students viewed paintings from 12 different artists with similar styles and they viewed them in either what they referred to as a blocked or a interleaved fashion. The blocked fashion, all of the artist's paintings as a group were looked at for a period of time. So the artist may have had five different paintings and they looked at the variations in that particular artist and they looked at another artist for five and off they went. And continued that process. And the interleaved, they randomly mixed the art paintings from the different artists as they did the study again over the same number of paintings and the same number of artists. But when the tests were done, the people that were in the interleaved treatment performed significantly better, being able to discriminate who the artist was responsible for the painting. And finally, I'll look at an example from Rohan Taylor from 2007. This is a college students who were looking at finding the volume of four obscure geometric objects, one of which is shown on the screen. And then completed one of two randomly assigned practice schedules. So you see the geometric object here a cylinder where the students supposed to calculate the volume of the wedge that cross-hatched area on the screen. The student is presented with the same practice problems where each group of students worked on the same practice problems but either the practice problems were blocked, four problems for one solid, four problems for another solid, etc. or systematically mixed. Both these mixers and blockers completed two practice sessions the same length, separated by a week, and tested a week after. That is the same old methodology we just described a minute ago. Set one interleaved, set one grouped, wait a week, same practice, same pattern, then do a test. The results, and this is what's really interesting here, during practice the mixers didn't do nearly as well as the people studying in blocked activities. Now that's challenging because that means the people who are doing that interleaved study, when we just described typically do better when you do the tests, are finding that their reinforcement in their study activity is very much less than the reinforcement that the individuals in the blocker group are getting. So when they did the actual test, you see that the performance on the test, their accuracy for the mixers, is much higher than those in the block group. And herein lies the conundrum. That is to say, we know something about the studies that really works, we know something about what works for performance and versus longer term retention. The problem is that we want to presumably address the issues of longer term retention, but the reinforcements and the affective experience of the learning process are telling the students that it is not working for them and not working as well. So that is effective practice is not intrinsically motivating. Now why is it taking us so long to recognize the superiority of some of these things like over learning or latent learning with repeated trials, frequent testing, the spacing effects, etc. And more importantly, why aren't we building these characteristics into the digital tools that we're using? Well, part of it is because the reaction of our students is not pleasant. Our students are not particularly happy with the fact that they're getting results which are not what they're looking for as they are studying. But they may be getting results in the final exams or the midterms, etc. that are better. It's just difficult for them to see the connection. And certainly when you're not feeling reinforced in the process. So they will choose studies that lead to suboptimal long-term results if we don't do something different as we're going through this learning process. This is what BORT calls learning patterns that contain desirable difficulties. And you've heard that phrase before. It's a big element influencing learning acquisition and composition and performance. And one of those activities that does that is this recognition of one's own abilities. And that's in terms of metacognition. So we know what makes things work. The question is, do we want better test scores or better learning? For better test scores, we know the things that will affect and improve performance. It's more reinforcing for students and it doesn't necessarily help learning in the long term. One of the painful things about our time, according to Bertrand Russell, one of my favorite scientists and authors, is that we feel certainty. Those of us, rather, that feel certainty are stupid. And those with imagination and understanding are filled with doubt and indecision. And there is a challenge for us. Metacognition, that awareness of our abilities or awareness of things around us, give us that sense of either certainty or uncertainty. It's our understanding of our own thought processes. Now, there's a long-running national public radio program in the U.S. by Garrison Keeler. And it's referred to as a prairie home companion. I'm sure you can hear it on the web. And one of his taglines is about the mythical community in northern Minnesota, a Lutheran community primarily, where he describes all the men are strong, the women are good-looking, and the children are all above average. That's actually a good example of a problem with metacognition. In fact, our college professors, my colleagues, have the same problem, because his survey back in 77 by Cross showed that 94% of them believe that they are doing above-average work, a figure that obviously defies mathematical plausibility. Now, this is where our first video comes in, and I will ask Martin to play a well-known psychology amateur, shall we say, who has a very useful description of work by a researcher named David Dunning in the U.S. at Cornell University about metacognition. Martin. The problem with people like this is that they are so stupid that they have no idea how stupid they are. You see, if you're very, very stupid, how can you possibly realize that you're very, very stupid? You have to be relatively intelligent to realize you're stupid. There's a wonderful bit of research by a guy named David Dunning who's pointed out that in order to know how good he is, it requires exactly the same skills as it does to be good at meditating in the first place, which makes this a terrifying fact that if you are absolutely no good at something, at all, then you lack exactly the skills that you need to know that you are absolutely ignorant. And this is going to not just hold you, but you're always going to be tired of your first image. So, I think that that's a relatively fair summary of Dunning's work. Dunning and Kruger back in 99 looked at the performance of students both in tests, and what they actually did is they asked the student to describe how well they thought they answered a particular question as they were doing each question, and then they asked them to place themselves, as when they were done, in a distribution about where approximately relative to their peers in the class, they felt the result of their test scores would place them. And what they found was that the top 25% that their skills lay in the 70 to 75th percentile, although their performance fell roughly in the 87th percentile. Now, Dunning and Kruger suggested that this underestimation by that top group of students was because the top performers find the tests that they were doing easier and therefore assumed mistakenly that their peers also found the tests easier, whereas the students in the lowest quartile viewed as they were giving their individual answers that they did reasonably well and placed themselves in the upper two-thirds of the class distribution when, in fact, they performed in the lower 25%. And this is true with students in class. They've gone out to look at people in real-world training environments, in sporting environments, the same kinds of things, and it's an incredibly robust finding and an incredibly depressing finding because it's also the case that very little has been found that actually results in interventions that can modify that misperception of one's metacognition. Now, some of you, I'm sure, are familiar with Serigo, but since I can't see you at the moment, I'm just going to assume that the... Can someone tell me whether there's a relatively wide recognition of what Serigo is? Hands up if you know Serigo. We have no hands. Okay, excellent. It's actually something that comes out of the predecessor of language learning tools. What I'm going to show you next, or Martin will show you next, is two set of clips. The first one is from the Serigo vendor describing what the tool does. And the second one is from its implementation in a UTX, that is the University of Texas, Austin, edX MOOC course by a colleague in jazz who used it as a tool. But the essence of Serigo is that it tries to define the memory decay curve unique to you for particular kinds of learned material. And thereby, by having, by doing that, it then uses that decay curve information to pull back information and represent it to you at the point at which from the decay curve it appears that you would be starting to lose it. So it's a way of reinforcing based on your own memory for particular kinds of things. So, go ahead and play the next video. Every time I study the Serigo, the system assigns its own strengths to each of your entities but it's done only through factors, such as last time seeing and continuing to perform it. This is the next select, which items are most urgent for that moment and grows less important with these items. This urgency value that takes place between all the right allows us to determine what information you most need to see at any given moment in time. And by presenting that information to you at that very moment, we give you the best chance of consolidating it as long term now. Okay, so you get the general idea of how the tool works. Now let's look at the way Jeff has embedded it in his edX course. To get started with Serigo, go to the subsection that takes you to the set you wish to set. The titles of these sections we deal with the word practice. And are followed by the title of the particular set you will be studying. Going to each section will launch the learning application, customized to your progress, directly on the edX site. Each set of items corresponds to the learning goals of the course. Serigo will first present you with the information to be learned and show you the correct answers for each of the items. It will then begin asking you questions that demonstrate your knowledge of the items. By answering questions correctly, you will make progress towards memory requirements. But there are much more than just questions and answers within the set. You will find vivid notes of each item accessed by putting the notes down at the top of each item. Like a system explanations of the correct answers, additional information about the topic, images and audio excerpts. So there you hear Jeff who is using, Jeff Helmer who is using that in his class as a means of trying to reinforce the particular attributes associated with the recall of musical styles with performance characteristics with characteristics of instruments, etc. And it seems to have worked quite well. Now you can get a synopsis. These are both the Serigo tool and a whole slew of other ones are summarized in a relatively recent March 2018 slide, which is another presentation from Ithaca. That's Ithaca with a K, this is the research group. On personalized learning and an overview of those technologies. And Serigo, Smart Sparrow, Leap and the host of these tools are among the emerging crop of applications which are trying to take what we know about learning and both in the neurophysiological sense in terms of memory decay and such, but also in terms of space practice and the like and build them into adaptive learning pathways. Separate talk about my concerns about adaptive learning in the context of social learning, but that's for another time. Now one of the reasons why these tools are coming out there and we're able to begin to take advantage of them is the emergence of an interoperability standard in the technology side called LTI. And LTI is simply a standard by which a tool provider and a tool consumer interact with each other in such a way that the learner does not have to do a series of redundant tasks in terms of identifying and authenticating himself or herself when the tool is brought up. So what's the reason I wanted to raise this LTI tool and both Serigo and Smart Sparrow and all of the rest of these things are available in LTI contexts is because we're starting to see the emergence of a dynamic and vibrant developer community which is being fostered by the ability to make an application which can plug into the LMS of your choice and by doing so create a market for experiments in the use of more effective implementations of cognitive science and learning where there is now a market big enough to be able to support these experiments. I'm not going to go into the details of how the whole thing works. We can talk about that afterwards if you like but this is really significant and therefore I encourage you to be thinking about this on your campus in terms of policies for the use of LTI tools and the issues that surround their effective and safe implementation. You'll have questions that you'll need to ask about well where does the data get stored? What's their policy for retention? What's their use of that data if they are trying to make use of it on the side, et cetera, et cetera but again, that's for another time unless you want to deal with that in a question later. Okay, so I talked to you about the metacognition stuff and as well as what we know about learning and how what we know about learning gives us some concern because the things that really tend to work are not well reinforced emotionally and in students' experience with those learning practices. But there is reason for optimism and this involves this notion of embracing a virtuous cycle in our teaching. This is very recent work by David Yeager, by Carol Dweck, by a host of researchers in the space that's often referred to as studying grit or various things along those lines and in the process, there has emerged this notion of self-transcendent motivations for learning. This is an interesting study that David and company has done, Angela Duckworth and a whole host of folks and it's addressing that problem we've just been describing. Many important learning tasks are uninteresting, they're tedious, they're boring and this tends to result in their being performed with less frequency and less intention than they should be even though they're absolutely essential. So this is a paper that I'm going to be talking about to the last part of this talk, relatively recent. The title of the topic is Boring but Important. Now, what they did in this presentation, they did a whole set of four experiments and what they did was they had students and these were high school students in this particular case that were presented results from a survey that communicated to them motives for why they might be willing to learn or wanting to learn, things like personal self motives making money, getting freedom and their ability to choose work. But some students, most students actually even if sometimes secretly are also motivated to do well even in spite of the social pressures in high school and do well in order to gain skills that can be used for pro-social ends. Survey statistics are presented to participants that indicate that most students were motivated to do well in high school, at least in part to gain knowledge so that when they can have a career that they are personally enjoying and satisfied by and to learn so that they can make a positive contribution to the world. That's a message that a set of survey statistics were presented back to these learners. The summary statistics were accompanied by representative quotes purportedly from upperclassmen at the schools reinforcing this social message. For example, for me getting an education is all about learning things that will help me do something I can feel good about that matters to the world. I used to do my school work to earn just a better grade and look smart. I still think doing well in school is important but for me it's not just about the grade anymore. I'm growing up doing well and preparing myself to do something that matters, something I care about. That is a quote of a student upperclassmen that these students got in a survey response. In fact, the survey response and these quotes are manufactured. They are simply created to generate this perception of a transcendent motivation. To facilitate internalizing this message, the students were asked to explain how they would help be the kind of person they wanted to be or help make the kind of impact they wanted to make around people and society in general. So they had to write two to four sentences in general and in this way, instead of being passive recipients of the intervention, the students themselves effectively authored the intervention. So they were allowed to make both the message as personal and persuasive to the self. The primary dependent variable in the study is remarkably of these are courses in STEM classes, science, technology, engineering and maths, where the grade point average is that the students got in the fourth grading period of the year. Assuming that low achieving math and science students might be more likely to be tested about the purpose of their learning and that that would have the greatest impact would be on those with the math scores that were lowest and science scores that were lowest in this pre-intervention. So they focused on the students that were doing least well in this intervention scheme. They controlled for race, gender, IQ and a sundry other variables that you might think would influence these results. And the results that they achieved are pretty astonishing. So you see here that on the left side you have all students and of 338 in this particular cohort and students with poorer grades. In this case the three represents a B in this particular grading schema. So things that people who got less than a B. There was a significant standard deviation, a lack of overlap therefore significant result for those students who had the intervention as opposed to those students that didn't. The point of this study, this is a second of a set of four is that by purposefully and articulating the learning goals and outcomes in a more transcendent way they could in fact significantly improve their performance. So does this purpose narrative that keeps the students focused on the study in the face of competing more desirable alternatives they actually then presented the same experiment but they gave students a whole set of boring math problems in a screen that allowed them to either do the math problems by going to the left side of the screen or had Tetris playing on the right side of the screen and they were told you can play Tetris if you want, that's no problem, that's okay, these are just practice study questions but they are useful for learning the material. And when they did that they found that those students that were doing those math problems in the control group they had a first set of students that were doing there was no difference in block one. Block one happened to be with no interventions whatsoever and in block two they had the intervention with that virtuous transcendent set of messages where they presented the students with the description of upperclassmen's survey results then quotes from those upperclassmen's that they consider being more capable of helping themselves and others and indeed the numbers you see there, 37 and 56 those are the actual number of problems they performed on the math side, so the people that did the treatment that was the virtuous reinforcement of transcendent learning actually performed almost double the number of problems that were the boring problems in the face of that context and they described them when asked about them in a different way. They actually described them as being well they weren't very interesting but I know it was important for me to be able to manipulate these equations in order to be able to learn the material for later use. So I want to give you some closing thoughts. I just want to re-emphasize how impressed I am by this set of studies. It's a huge study. I suggest you take a look at David Yeager's work and it gives you some really positive, optimistic suggestions for ways in which we can motivate students to do the kind of things that we know work for longer-term learning but which are not reinforcing necessarily by just presenting the rote activity for them. So closing thoughts. New digital tools are decomposing integrated learning systems and that's one of the reasons why that LTI environment led to that explosion of new tools that allow experimentation to actually be tried out in real world contexts with enough of a market for them to be supportive. The LTI issue on your campus is something you need to really be thinking about. There's a caution. One of the very common LTI tools out there is something called Piazza for discussion. Discussion boards and such. What you may not know is that the back ends data that Piazza collects on the students put their posts in the discussion board are actually mined and sold to businesses and that's their business model. That's how the Piazza tool is free. They have in the US refused to sign the FERPA agreement on federal privacy protection of individually personally identified data and at least our institution is banning its use. So we talked about adaptive tools. I'll give you an example of one of the set of adaptive tools and personalized learning tools and the question I raise there in terms of where you stand on that is because personalized learning tools are very much oriented towards the individual learner and the question that it raises is are we creating a learning environment that Sherry Turkle has called being alone together, ignoring the value and the importance of the social context of learning by creating these personalized learning pathways which have very effective or potentially effective implementations of cognitive science in the actual way in which learning is presented but in fact may not take any advantage of or the capabilities of the sociality that we know is important in learning as well and so that's something to be thinking about. This leads you to of course take a step back and ask what is your ideal learning environment? What is your theory of learning? And of course the classic from James Garfield was two people sitting in a hut one on either side of a log and the instructor at one end, the learner on the other and they're just having a conversation and that is this particular idealized view of learning it's a very romanticized view of learning and one in which I think many faculty actually ultimately have but which isn't necessarily helpful in thinking about how their students perform or the kinds of exercises that need to be designed to make them perform better and finally in the face of this expansion of the scale of our institutions that is the size of the number of students we have to turn to teach, etc. University of Texas at Austin has 54,000 students residentially on campus. What are we doing to bring good learning science into elegant learning environments that fits our culture? And this is of course the most challenging aspect of this a culture at UT is relatively conservative it's relatively focused on providing students with an instrumental learning environment and there are of course examples of progressive work and such but I would say that lecture still holds a particularly dominant place and that's in part because it's a very research intensive university and we know as a lecturer, as a faculty member how to do that type of teaching and get out and go back to our research which is where our rewards come from. So that's where I want to leave you with the notion that we know a lot about cognitive learning we know a lot about the ways in which the process of doing so is effective and not necessarily individually reinforcing for students as they're performing it we know we have to think about ways in which we can address that some of the adaptive learning tools are trying to do just that and more importantly some of the most recent research by people like David Yeager and Carol Dweck and Angela Duckworth and others are suggesting that there is a possibility of using metacognition in a way we weren't really thinking about to change the perception of students about their own motivations for learning by engaging them in thinking in a context which they actually bring to the table but we don't reinforce and with that I'll say thank you We've lost him We may have lost Phil but I hope he had the round of applause we're now seeing whether we can reconnect for some questions that would be unfortunate if we've lost him He had this problem when he ever closed PowerPoint it would crash his hangout for some reason I guess Google and Microsoft weren't playing nice Perhaps they were listening to him Right We'll have to see what they're hearing Perhaps I can put you on the spot Amanda We do have a mic so if people want to present you you've got to close it off Yeah Here he is We've got Colbert Hi there I have no idea where I left off You got to the very end It was wonderful Remarkable Fine, thank you We were just applauding and you disappeared but hey it's good to have you back That was not meant to be a statement I think we've got time for just a few questions Do we have a couple of microphones going around Comments or questions for Phil Can you hear me? Yes I can You started on the very good question about how often we're not using what we know from learning research in the way we develop and design learning tools digital tools and environments and then towards the end you kind of often answered my question because you were talking about the way in which some of the studies don't address the social aspects of that Of course that is a very important part of the cognitive learning that we do The kinds of studies that you've quoted I mean the latest stuff I think is probably different but some of the earlier ones are such piecemeal types of learning need to be texturized not the kinds of things that people in our kind of field are kind of want to or interested in working much more on the real world of what students are actually trying to learn in the context of their degree program So I'm just wondering if you could comment on the types of studies that we shouldn't be paying attention to Would you agree that there are some of those that aren't helpful already? Some of those that are say the last bit again but aren't already Yes, I mean So I think that those the things that you described as piecemeal or decontextualized are still important because the actual demonstration of a particular phenomenon or effect are things that we are that we need to take into account when you bring the student into the richer interactive more complex learning environment I mean just for example that notion that studying in multiple locations facilitates acquisition as opposed to studying in the same place in the same way over and over again and then putting yourself ironically in a new environment for the actual test I mean that's something that on the one hand you could say is a very sort of decontextualized carefully controlled experiment but one which actually says a lot about what we advise our students to do when we're asking them to prepare for exams and things and that would be actually when you do your study go around to different places put yourself in distant situations find out where the exam might be and replicate that environment when you're studying as well because you'll find you'll do better so there are certain things about those decontextualized studies which I think do still have context but nevertheless I think it's a very good one. There are people that are trying to be much more real and and much more contextual in the sense of providing ways in which students can be given guided learning experiences that pertain to actual uses of those of that learning in particular ways that they will practice potentially in the future which is why we are so interested in for example active learning and performance particularly in the STEM fields that give students an opportunity to engage in the kind of learning thinking and study experiment environment which we think is where their discipline is taking them and doing that as early as possible because we know that context of that learning in that way even though they may not be able to perform at nearly every level of others who have been practicing in that context for a long time nevertheless that context has all of those other capabilities around the latent learning that's going on the social interactions that are happening in that context etc which will be extremely valuable Duckworth stuff David stuff David Yeager stuff etc I think are really trying to keep that forward thinking embedded in the learning environment of the real world at the forefront and nevertheless engaging in some very point interventions to construct a framework of thinking while the students are engaged it's just remarkable to me how much of an impact those things have had in the experiments that they've done Thank you Thank you Have we got another question comment Yes Just getting the microphone I can see I can see Good Thank you That was so interesting I was wondering about community science and other disciplines than some of the ones that depend very heavily on memorabilia nation So some of the as the subject has related to mind and to each other and since it's like a new idea or coming up with critiques what do we know about those problems of science There was enough echo that I'm having real trouble deciphering that can someone repeat the question directly into the mic perhaps Amanda could you repeat it No No Amanda just speaks it into the microphone Right Can you remind me it's the cognitive science Right So moving away from the STEM subjects into those which embrace more of the humanities and social sciences What about cognitive science and its applications there Well, so for example I mean the briefing the briefing yet that you saw from the jazz class sort of fits that where they took advantage of a series the way that tool works it gives the students a series of questions and tests along different aspects of music in this case and then tests them over a period of time and the process draws for every individual its own memory decay curve associated with topic areas of questions and then it uses that as the trigger for the reinforcement cycle for those kinds of concepts as the course progresses So that's an example outside of the STEM sciences but I think the other areas that you're thinking about for example I'm working with someone in the theater and dance program here who is using drama and dance as a pedagogical motivation for training teachers I think that there's a lot we can understand about the engagement that learners that cognitive science has in terms of the connection between proprioceptive and body movement and memory the relationship between affective states that are engendered I mean the whole drama based instructional paradigm is around the notion of heightening particular emotional associations with learning and that connection is something I think cognitive science has a lot to say about one of the things we want to do is actually instrument her as she's teaching so that we actually get a three dimensional representation of her movement and of her voice modulations the decibels of her voice and things like that to be able to see whether we can distinguish those patterns and their relationship in students that she's engaging with and make a connection between things that we can measure about her activity and performance and the learning that the students are gaining or not in their interactions with her and can we relate those to other attributes of their psychological characteristics whether that's the person's mindset for example if you believe in the Dweck work or self-efficacy or other characteristics for example okay thank you we probably have time for another question or comment we have one right at the back right at the back I will try to write it down this time yeah it's the echo that I'm getting killed by hi I have a question basically it's about student engagement really we're entering a period where we're increasingly involving students and that's usually mainly about their learning and obviously probably the work that you just highlighted here we're in a situation where actually what's good for students in terms of learning is not the sort of thing that they're able to do for results initially or they're under cover how then when we're involving students in decision making do we communicate the benefits of doing things in a particular way yet acknowledging their voices at the same time okay I think I got that I think I got that one this is really interesting because back in 1922 Albert North Whitehead did a presentation actually to a training council in England where he introduced this notion of the rhythm of education and talked about the first stage of it is romance where you're engaging in a topic and it's exciting you don't know a lot about the details of it but the prospect of what it might mean for you down the road and it's the important questions that the discipline offers you are enticing and then we end up, he called it going through a stage of precision which he talked about which is kind of like getting the procedures and the details and the facts and the fundamentals well understood so that we could then do what he called the third stage and the final stage which was generalization which is where we can apply it to different contexts and actually take advantage and use it and of course from a student's perspective you can imagine in that particular model that the romance in that first introduction to something new is exciting and perhaps frightening but then we get slammed by as a student all of the practice and all of the prescriptive information that we need to understand and the rote repetition and the building of the facts and eventually maybe with a good stuff at the end we'll get to so I think if I understand your question correctly one of the things we have to do is we have to bring the good stuff in periodically from the beginning I mean all of us who are instructors we're in our disciplines and doing things because there's really exciting fundamental aspects of our discipline that we care about which we think are really really important the problem is is we tend often to think that you have to have this foundational knowledge of prescriptive facts and skills and the like before you can meaningfully engage with that good stuff the things that really are important or have a real world consequence and the reality is that that's not the case we can in fact introduce those complex ideas even if we know the student won't grasp them entirely and we can in fact are we still on I still have your video no okay that is going to go online so it's going to be really romantic but I think that you do have four minutes to fill and to fill up your questions and you really thought through those big ideas but also most of us and also I've been trying to speak to so really personally able to take the opportunity for their support and for us as chairs it's really important that we think that just a practical I'd just like to add a personal thank you to Amanda this was her idea she said this I want to share obviously could you share it with me I'm not thinking I'll say yes I don't think it's a good idea but I think it's a good idea thank you so much for thinking it's okay are you it's been great for me and so a special clap for Amanda please I think it's actually important that we are learning together but as ever which I've come up to speak totally unrelated around the thought in mate to the order and one that struck me was actually the great thing about autism it brings the sectors together we can have as a big evidence yesterday and it's great as it seems across the nation across the sectors and so I think it's really important that we think about students as innovators so that's one great thing that's happened second thing that I've found ideas around inclusivity so that the micro level of course design by these students experiences and invited their participation through their own faculty is the idea from presentation about those students in Chicago and also the need for early art on the risk of telephoto-appropriate or in fact making another presentation I think that was a joke and also of course at the macro global level there was very hot felt and well thought out community so it was slightly a reason for participation and a lot of coverage between universities and the global world and the south and participation are intertwined and don't confuse and also attach as well but it's extraordinary to have workshops with these ideas which of course to go away and try something out in our institutions and sort of to practice with these community students but I don't know I think the evidence in the room but there is an elephant and we need to guard against easier sanctions to necessarily embrace change in innovation and may have to work with it to take them with us so those might be emerging from the conference in short I think we will be developing ideas about it over the next few days and they sound to do something wrong to some of my old students and Larry just said my four year old asked me to bring her to go in and talk about this this is my student Edelman Martin who made a new conference to talk about the amateur in the war in the Sackville Park who is with the reflection there now let's see the final one was today the law who has been continuously online according to my research it's been realised that the future is hard to predict if you've been able to see that all is supporting the OENR16 which is a popular conference and I would like to explain the way that we've done obviously in 2016 we were standing in the wings we asked Michael and Jess depending on who they were trying to actually I think we do as well for this year we want to say we want to say on this stage our teachers of this year our teachers of next year and our games organiser Julie Rosie and all the way from the background to our switch equipment and all the cables to come up as well all together one more time again this year has been an absolute pleasure and it's done a great job but I'd like to particularly recognise across the region as the artificial children account the lectures and the expressions comes up for three days and my only question is how can I find an organiser and also for the games organiser I think you can see a quite broad start looking at it which you can't for sure but we've done three fourth ones so yeah and again in between the privilege it's done three days with you and then together and just a quick note for the last three days we've done a great job and it's been a safe journey so one more time please say thank you to everybody and to yourself thank you bye