 Okay, so thank you all for coming here. Now, I'm a technologist. As Mike pointed out, I've been a teacher and I've been a lifelong learner. But before this meeting, I had no real contact with the official learning technology discipline. And so I was rather apprehensive when I got the invitation from SEP to come here and talk. I said, gee, you know, before I could do that, you have to read 10 or 20 years worth of research papers just to catch up, to be able to talk to you guys, and that seemed kind of intimidating. But then I was very encouraged when I talked to this friend of mine, Hal Appelson, a professor at MIT, an educator, author of probably the world's best book on computer programming. And he told me, you only need to read one paper. So that was a big relief. I figured that was a reading assignment that I could catch up with so I was able to accept SEP's invitation. And this is the one paper that Hal told me to read. It's a paper by Benjamin Bloom from 1984 on what he calls the two-sigma problem. And you probably can't read that very well. But what he points out is a comparison that he and his students had done of three different learning methods. So the first is what he calls conventional teaching in which students have the subject matter in a class with about 30 students per teacher and the teacher lectures and gives texts in the way that we're used to. Then he compared that with a model of Bloom's called mastery learning. And there the idea is you do the same kind of lecturing, but instead of having tests to separate out the good students from the bad students, you keep on testing until the students actually get things right. And so you test to identify what their errors are, then you go back, re-teach them on that and have them keep going until they learn all the material. And then so the second approach is more teacher intensive and more personalized and therefore more expensive. And as Dylan Williams was pointing out in the previous keynote, this is something that perhaps we don't have the resources to do. Maybe this is too expensive for us. And then the third approach is the one-on-one tutoring approach, which we know is too expensive. The results that Bloom came up with in this study was that there were significant effects to both these augmentations. And he says you get about one standard deviation of improvement from using the mastery learning and two full standard deviations from using the one-on-one tutoring. So according to Bloom, we have the answer to how to make our students succeed. It's just that we can't afford it. And now the challenge is, is there something else that we can do that's going to be as effective as one-on-one tutoring that uses some other kind of technology? In 1984, we didn't have all the technology that was available to us today. Is there something else we can do now that can answer Bloom's challenge from so long ago? So first I want to talk a little bit about what exactly conventional learning is like and then contrast that with some of the other possibilities. So I learned a little from another educator, Frank Rhodes, ex-president Cornell who said, in the basic business of teaching resident students, universities have not diverged much from the method of Socrates, except that most faculty members have now moved inside. So what do we know about Socrates? Well, maybe he had reasons for thinking that being inside could be a corrupting influence. So he did indeed like to teach outside. And we know that he's famous for this questioning approach, that he used questions in many ways. One was to keep the inquiry focused on what he thought was the lesson to be learned. Another was to keep it appropriate and personalized to a learner. Another, he used it in sort of a bullying way to kind of show the learner that he really knew what was going on but wasn't able to tear it out himself. So maybe some marks off for harassment there. But it was a different time. But note that in this one-on-one questioning interaction, Socrates is really much more like a tutor and not like a conventional teacher. So Rhodes was really wrong. It's not that we haven't moved forward at all from Socrates over 2,000 years. It's that we've moved backwards from this tutoring approach to a non-tutoring approach that's not as effective. So now let's fast forward to Millennium and a Half. Here we are. This is conventional learning in the year 1350. Now the lecture was a fantastic innovation in 1350 because books were very expensive things that had to be made by hand, by monks. So you couldn't really afford to give them to all the students. But in a lecture, one person could read this material and get it into the minds of all the students at once. It was the broadcasting of the time. Now you'd think that although it was a great technology for 1350, by 100 years later, the lecture probably should have died out because there was a new learning technology that this guy Gutenberg came up with where it became cheap to have these books. So why have this one guy in the front of the class reading from it when all the students could be reading from it simultaneously? Now I'll leave it to the historians to sort out why that didn't happen and why the lecture didn't die out in 1480 or so and I'll concentrate just on the problems with lectures and note that we don't need any of today's technologies and studies although there have been plenty well-documented studies. We can just go back to 1350 and they've identified the problem right here. It's this guy. So even in 1350 we knew that attention wanders when you lecture. People fall asleep and you can look around at the other people in the room and only about a third of them are really paying attention. The others, if they're not sleeping, they're starting to nod off or they're talking to their neighbors. That's what happens. Now in places where educational results really matter, regulations are actually put in place to limit lectures. So for example, in the United States Federal Aviation Administration is responsible for the teaching of pilots and they put in regulations that say no lecture can be longer than 20 minutes maximum. Why do they do that? Because they know if they don't, that's what happens. So they found out that there were bad results and they put regulations in place to take care of that. Unfortunately in other industries where maybe a lapse in attention is not as dramatic, the lecture survives unregulated. Now I notice that this particular enlightened conference for most of the sessions does respect the 20-minute limit. Unfortunately there is one exception to that. That's the keynotes. So that bodes ill. We'll just have to hope for the best. We're only about eight minutes into the talk so maybe catch about five minutes more than wander out, talk to your friends and come back at the end and I'll try to stop early and leap plenty of times for a more effective method of interactive talk. So that's one of the big problems with lectures is that you just can't do it too long or it starts to get stale. There's another problem with it the lecture is suited only to the oral learning style and now we know that there are multiple methods of learning and as an aside I was glad when I went to my daughter's third grade parent introduction night where the teacher explained what was going to go on in the classroom and she started to talk about these different kinds of learning styles and the teacher said, well you know we recognize that some of the kids are visual they respond to written words and diagrams some are oral they respond to spoken words and music some are tactile they respond to body movements and touch and we try to individualize the instruction to be appropriate to each child and the modality that they prefer I thought this is great here's a teacher that's making use of the latest research my child's going to get a good education and then she went on and she said and in fact you can diagnose which type of learner each student is just by listening to how they talk and she says you know if you're a visual learner you'll say things like I see what you're saying if you're oral you'll say things like I hear you and if you're tactile you'll say things like I feel that you're right and I was thinking nah you know if it was that easy but that can't be right and then I was heartened when Google hired this guy TV Rahman who's very well noted and respected engineer happens to be blind and within my first five minutes of talking to him after he was hired he said nah I see what you're saying so I think my intuition was right on that one and it's still as difficult to diagnose exactly what learning style a student is using and they may prefer different ones at different times but it's right that you don't want to rely on a modality that just goes after one of those styles so that's another problem with lecture now one more thing about the lecture or the conventional approach is that it lacks portability portability and let's compare it to another form of delivering information the musical performance now back in 1530 if you wanted to hear a lecture well you were lucky if there was one place to go to hear a lecture within a day's hike and that's what you were stuck with now if you wanted to hear music you probably had a few more choices but basically it was whatever troubadours happen to be in town you could go and listen to them perform music or you could perform it yourself with your friends and family now in the modern world of course here in the 21st century if you want music you're no longer stuck with your local troubadour you have thousands of choices and you get the very best performances by the very best musicians in any genre you want replayed at any time but contrast to that to what's happened with lecture so if you want to hear a lecture up until the very final years of the 20th century you only had the choice of the live performance you had to hike down to your local performance place which is the university and hear the lectures there now of course in the last few years that has become a little bit more portable and you can shift in time the lectures from your local university you can even get some lectures from universities around the world but we've not moved to this model where there are star performers you know we could have imagined a model where if you want to learn about linguistics you said who's the best linguist or is this guy Pinker I'm going to go get him put him on my iPod I'm not going to be stuck with whoever happens to be at the local university but for some reason we didn't go down this route and I can't really explain why okay so let's look at which learning technologies might help solve the two-sigma problem so we saw the problem with the conventional approach with the lecture-based approach how can we get away from that towards the tutoring-based model without all the expense of hiring a separate tutor for every student I'm going to talk about some of the things that Google is doing in this area as representative of the kinds of technologies of course this conference has covered many others and so we're going to start off with the sort of more formal approaches and then move to the informal and think about why they may actually be a better approach okay so first thing is Google is offering these Google applications education edition so that's free email, chat, calendars, web pages and documents and branded with your university your educational institute and help in putting it all together and up until two days ago I thought that this was just a collection of online applications but now I know that if you integrate it all properly it's actually called a virtual learning environment okay okay now here's a little sketch on the back of a napkin that was drawn by Adrian Sanyer who's in the IT office at Arizona State University and he called this like technology for an advanced alien culture and so what did he mean? Well he noticed as you see on the left here that there was this increasing gap so he as information officer at a university said you know we've got limited budgets we can only spend so much and it seems like every year the best of industry the highest technology gets farther and farther away from us and we can't spend enough to keep up but he said but if we move to this model on the right where instead of us trying to buy our own technology every year we ally with a partner now we're suddenly we're on our partner's growth curve rather than our own growth curve and we can go as far as they go and see what approach he's taken you can see there's lots of things that come with this you can get help with so you can there are tools at the school level like setting up a website and there's tools at the classroom level like planning a lesson using Google Earth Maps and this is in a partnership with Amnesty International one of the lessons plans we provide and there's many other types of lessons available okay now here's another partnership and this is fairly new just launched last week we didn't talk about it much but we've teamed up with Creative Commons and with the Packard Foundation from Hewlett Packard to create an index of all open courseware so any university or other institution that publishes their course on the web can find their way into this index and we'll search it and give you back the results so here you see a search for solar energy you get the MIT course first that's appropriate they're good engineering school and they're at the forefront of this open courseware movement so maybe they deserve to be first then there's other results from world watch from other schools and so on a combination of different results that you can browse through and see these educational materials and this is useful both for the teacher trying to put together a new course of their own and for the students trying to supplement the course they're taking and for the home learner who wants to put together their own course okay so that's one option many of you probably familiar with scholar so here's an opportunity to search over academic journals so if I search for ivory build woodpecker I can see a recent journal article from science that caused a lot of excitement I can see an older book that talks about some of the background behind it and so on collection of articles but some of our users are not familiar with scholars so this is not a destination that they would go to it's sort of hidden to them or not available to them and so one of the things we've been doing recently is trying to bring all our results to all our users without them having to know where to go to so if you know to go to scholar great that's what you want you're in good shape but if you don't know we still want to get you the right mix of scholarly and non-scholarly material so here's an example of a search using the normal Google search for big table which is implementation that some of our engineers have come up with for a very large type of database table used over parallel machines and there's a mixture of results here so there's papers that would have been in scholar these academic papers there's a Wikipedia entry and the third entry there there's a video and then some blogs and then some more academic papers and so on and the point is that if you thought about this on your own and tried to narrow where you were going to search you probably would have said well this is an academic type thing I'll go to scholar I'll look there but then you would have missed the video and some of these other entries you probably wouldn't have thought first that video is the kind of thing that's with this kind of query but in fact we have the engineer who designed the system giving a technical talk and that's probably exactly the kind of thing you want so in some cases you can be outsmarting yourself by going narrowing yourself down to a specific search okay and of course regular Google search works pretty well as finding subject matter for course as well so all those things are available and they're available for the teacher formal student and for the informal student now let's get back again to Bloom's challenge and remember what he said that the average student under tutoring was two standard deviations above the control and try to understand exactly what that means because it really is quite extraordinary so standard bell shaped curve the control by definition 50% are above the mean if you define things just right and you can say that being beyond two standard deviations means doing really well then 2% are doing really well now when you move to mastery learning things are shifted approximately one standard deviation to the right you can see the outline bump there getting a little bit of separation from the control but maybe not that much but still quite a change in that now 84% are above the control mean and 16% are doing really well so you multiply by 8 number of people are doing really well once you go to tutoring two standard deviations now you really got a big advantage you can see the two humps are widely separated 98% of the students are above average and 50% are into this range of doing well so that's an extraordinary accomplishment if it really holds up some more numbers from the paper in the control the conventional teaching students were on task 65% of the time so essentially one third of the time their attention span was wandering if you believe the research paper represented by that 1350 painting was more like two thirds of the time attention was wandering so I'm not sure which one to believe but you switch to mastery learning you pick up a little bit of that time back so now 75% time on task tutoring you've got 90% so you really got their attention and maybe that's where most of the games come from Bloom-Hoss reported these correlation numbers so in conventional teaching there's a .60 correlation between the aptitude so that's before you give the course you test them for kind of background knowledge and the achievement the results have taken the course so that's saying 60% of the result is determined before you even get started you're just kind of allowing the top students to stay at the top as much as pushing them on with mastery learning the correlation goes down to .35 and tutoring down to .25 so in part that's saying everybody has a fairer chance to advance in part it may also just be sort of statistical artifact because if everybody is squished off to the right then they all have about the same score so now the correlation has disappeared because everybody's doing excellent now in the paper Bloom had four main suggestions on what you could do to achieve these two sigma without tutoring one he said careful review of previous material for example teaching an Algebra 2 class make sure you go back over the Algebra 1 material and do this with what he called enhanced cues participation and reinforcement which is just a sort of sneaky way of getting the tutoring back in it's going over these materials seeing who's getting it and not getting it and if they're not getting it giving them special attention he also said a student support system which groups of two or three students study together work well of course this was back in the days when everything was done offline may be more effective now done online special programs for reading and study skills a sort of remedial program and he said that computer learning can work but only for very motivated students and maybe that has changed now as well so let's look a little bit at what kids today are doing really well or out there on the right past the two sigma mark and there's a number of ways you can achieve this kind of level of high achievement so some kids do really well on computer based training materials and this is a slide from this site called Hey Math and they provide this computer based training for Algebra and Dylan William talked about a similar program at CMU he said the results are very effective but it took 20 years to develop and so there are a few subjects like Algebra where it seems like it makes sense to put in that 20 years effort or every much effort it takes because you want it's complex material but it's very self contained and you can sort of put together the materials once and for all and really nail it and then you can teach it to a lot of students and so I think there are these few areas where that makes sense but for most areas it probably doesn't make sense and even for things like Algebra you have to think about how well these materials you put together are going to port to different kinds of students so for example this lesson it seems like it will work well for somebody who is a football fan but you know what about this kid and say he prefers baseball or this kid who likes cricket better or this kid doesn't actually know anything about organized sports but he's been hurting and counting his family's sheep since the time he was 4 years old so these are all very different types of learners maybe trying to fit them all into the same funnel computer based training with a specific theme is not going to work that well so what other kids are doing well so here's a kid who had a good summer with him over here but in the states he was hot so this is a 17 year old high school kid who in his spare time over the summer unlocked the iPhone traded the result for a car and then packed off to go out to college and he said he spent a few hundreds of hours and he had the help of a few online friends so this is just the exact opposite of computer based Algebra training there's no lesson plan there couldn't be a lesson plan because before this guy did it nobody knew how to accomplish what he tried to do maybe the Apple engineers did but they weren't telling there's no real theory behind it it's all practice and I guess actually there is some theory but he learned it as he went along so when he started out he was pretty handy with a soldering iron that gave him the confidence to get started he got into it I don't know much about that but I got some friends online and maybe they can help me and maybe they can lead me in the right direction and together they started to work on it he learned what he needed to know it was collaborative it was just in time learning and again it comes back the tutoring was sneaking in he was being tutored whether he knew it or not by his friends they taught him about software he taught them about soldering and then they came up with the answer so that's a successful result is it duplicatable? well I don't know is he an exceptional kid or was he just lucky or was it just the determination to get started on a project that was important rather than any attributes that he had to have I don't know here's another example this is Olin College so this is a college that's really dedicated to every kid this kind of iPod like experience so it's a new college they just graduated their first graduating class last year it's an engineering college located in Boston and the idea is that instead of theory heavy lectures and segregated disciplines and individual efforts they go after design exercises and interdisciplinary studies and teamwork and the curriculum integrates to these disciplines with practical projects so it's a do and learn approach President has shown here President Miller said students start out with an audacious project which would in many institutions be heretical except we do that deliberately because after all when you get hired in a corporation that's the first thing that happens to you they give you a challenge for which you've not had the prerequisites and it's about learning how to learn so we do that here from day one so this was the first assignment for the freshman class coming in so they divided the students into small groups and they assigned them the task of in eight weeks designing, building and demonstrating pulse oximeter so that's this little device that you clip onto your finger and it tells you your pulse and it tells you your blood oxygen levels without pricking your skin and so the professors showed the groups a commercial unit and then they referred them to some patents and some technical documentation and then the faculty's plan was to watch the students and see where they failed and stepped in and guide them towards a solution and the problem was they never failed they just got started they said we understand what this problem is we understand what we don't know and they went off and they tried to learn it in eight weeks all the teams got the project done and one of the side effects was that at the end of eight weeks they were now receptive for this theory that they didn't have so in other classes you would have spent two years learning about the theory of transistors and so on before they let you do a project here they skipped all that threw them into the deep end of the pool said go ahead and do the project they learned just enough to know at the end of that eight weeks they said you know I'm really interested in transistors now I want to know some of that theory and then they started to teach the theory behind it one more example this is probably cheating because this is from science fiction not from reality but this world that Werner Wynge imagines takes the Olin College to an extreme so this is a world in which in a fictional high school students are doing the equivalent of a year's worth of graduate level research in a few weeks time and they do this because this is a novel of the near future the students are permanently wired in they're sort of wearing their wearable computers and on the grid all the time and communicating with their fellow students nearby and across the world and sort of doing internet searches by blinking their eyebrows without having to type on anything and seeing the results projected and in this world many of the adults who are coming from the present day into this world of maybe 20 years in the future are lagging behind and so the adults are sent to high school for remedial education because they can't catch up with these high school kids so a possible future that way. Let's go back to Frank Rhodes and he has some answers to this question of how important really is this social networking aspect so is it just if you put the kids in connection with each other and with some potential teachers is that all they need or is the formal learning a vital part of the equation. Rhodes said that without community knowledge becomes idiosyncratic. The lone learner studying in isolation is vulnerable to narrowness, dogmatism and untested assumption and learning misses out on being expansive and informed contested by opposing interpretations and refined by alternative viewpoints without community personal discovery is limited. Okay so I think it's clear that this is important but the question is can it be the only thing or can it be the main thing? So I think now we're ready to answer this question, a challenge from Hal and from Benjamin Bloom of what education should be like how we can achieve this 2 sigma effect. Let's try to synthesize these examples and here's the best I've come up with. So I think the best way to get to this 90% time on task, individualized learning is to concentrate on these points. One, center the education on engaging real world projects. So the kids approach, their feelings about the project are probably more important than the actual words that are coming out of the teacher and if they're excited about it, they're going to get better results and it seems like a project based approach is the best way to get them excited. Exploring in teams seems to help. We can't afford individual tutors but we can afford to put people together into teams. You get that for free and you get much of these effects of the 2 sigma effect of the tutors just by having people work together and help each other. So now the teachers are serving as facilitators and they can point to theoretical knowledge when it's needed and part of the point here is that it's needed much less than you think that means that the teacher now is free to become a tutor because the kids are doing most of the work the teacher is acting as a facilitator and so one teacher can be spread out over 30 students and still have enough time to cover them all. And finally, different students learn differently but let them figure it out from the world full of information. Don't try to create all the materials ahead of time and make one set of materials for everybody to reach. So we've gone through these transformations in the way information has been used. We started off with a lecture where there was one copy of the book in the world and the lecturer was reading it, the students was learning it. We got the revolution from Gutenberg that now the students all could read the book and more recently we've had the revolution where the students now can do two things. One, they can access all this information from around the world all the books and all this other kind of information but more importantly they can access each other, they can access fellow students and other people throughout the world. We put that together that's a revolution that I think comes the closest and it's the only scalable solution to Bloom's challenge. So thank you and I'll stop lecturing and make the rest of this interactive. Thank you much indeed Peter. We do have time for questions and I hope that you're willing to take questions now. We've got a roving mic here and while you're thinking of what questions to ask, what we'll do is start with one of the questions that have come in from Illuminate and it's from Panos in Edinburgh Web 2 technologies seem to be very busy with social spaces and not necessarily educational learning places. It appears that it makes students to be busy with things not related to learning but only to information sharing. Is this education? Any comments? So from my point of view the point of these technologies is to connect the people not to be the only place where they connect. So once you found a site person that you want to work with you can do that in person. If you happen to live nearby you can talk on the phone you can use Web 1.0 technologies of email or whatever you want and I don't think you need to stay within the bounds of this social networking site or wherever you start it. I think it's making the connection that's important and what you do with it then there's many options. Other questions? Yes, there's one there in the centre of the hall. This is sort of coming back to the Bloom paper which is sort of one of my little hobby horses. The at the end of the Bloom paper he suddenly starts talking about well, what about higher mental processing? He sort of starts to realise that all of the studies in the classroom that he was doing was on students learning how to do things that they're taught the exact process, step by step of every single thing they have to do along the way and you also discover quite how bad his control group was because he finds that setting homework makes half a sigma improvement. Marking the homework makes another improvement actually teaching them stuff based on the stuff that you found out makes another improvement. Lots of educational technology things like the cognitive tutors that Dylan mentioned yesterday focus quite heavily and shown learning games by doing the Bloom-like stuff where you train people in a set process. Something that very few systems seem to do is let students try out their own ideas and kind of react to that and sort of show them the consequences of their ideas and it seems to me that's the only way that you can teach someone in situations where you don't already know the process and for most interesting questions you wouldn't already know the exact process we'll need to go through. So anyway, it was sort of just something that struck me. I wanted to make that point and see whether you had any thoughts on it. Yes, so I certainly agree with you and you know I went through the same process I read the paper and at the end I said you know this seems like it really applies to this kind of rote learning but does it apply to more creative learning and I said rats, Hal Abelson lied to me I probably have to read more than one paper. So I was concerned about that but I think you're right that I think you know the important parts of learning are not in this rote learning. I think Bloom did emphasize that too much and I think we want to move away from that and I think that as you point out, as you do that like the theme of this conference points out you're losing control. You're no longer saying there's a narrow path that you can follow and we know exactly what the right and wrong answers are and if you turn right or left that's okay because other people have gone down that path before and we know how to follow you up. You start to get into areas where there are no black and white answers and where the student may be exploring something where we haven't seen the answers before and I guess part of my message is that you should embrace that rather than fear it and be willing to go there and trust that the students are going to learn something it may be something new it may be something you as a teacher don't know about or have to figure out as you go along but go ahead and let them take that path. Another question we've got one down here while the microphone is coming down we've got another question via illuminate from Adrian in Manchester referring to your creative commons search engine I'm wondering what criteria we use for ranking the creative commons open courseware materials the one where MIT came at the top. I can't tell you the details I can tell you it was built on Google custom search which is a facility that we offer to anybody anybody can create a custom search this one was done by this partnership with Creative Commons there's two main things you can do one is you can rule sites in or out you can say I only want to take results from the following set of sites that meet these criteria or I don't want to take results from these other ones and the other is you can influence the core Google rankings up or down by saying the following things are important to me as I rate these results the tools that we provide now I have no idea what this particular search is done in terms of how it's used those tools to provide the ranking are the criteria generally set by Google or in consoles how does that work? the criteria are set by whoever publishes this custom search we give you the tools in which you can state the policy that you want to implement but it's up to you to state that policy I'm interested in the star performer idea and why that hasn't really taken off and what might help too or is it perhaps an outdated notion I'm sorry I missed the beginning which idea? The star performer and why there aren't star performers that you can get to easily I'm not sure why that is it just seemed odd to me that in music we've gone so far in that direction I'm sure you can still go to your local live performances but most people most of the time they got their headphones in and they're listening to the same top 40 groups or maybe they're not maybe they have very very choices but they're listening to this recorded music rather than seeking out the local one I don't know maybe the universities had a lot of power and they kept things that way until very recently I think it was very clunky to try to review these materials online it's still it's better being there in real life you get the chance to ask questions and so on but I've been in lecture halls that were twice the size of this one and there's not a lot of interpersonal reaction going on there between the audience and the professor so I'm not sure that you give up that much by putting it online and of course we could go so much more beyond that so we could have if you are going to go to this star model you could have much higher production values and put in animations and simulations and so on mixed in it's expensive to do that and it seems like it's probably worthwhile doing but it hasn't happened and I don't know why the start of your talk was some amusement because you used two exactly of the same images that I used in my own professorial inaugural lectures which was the image of the people asleep in a lecture theatre and the Socrates wandering around outdoors in my lecture I used a third image which was a group of neanderthals who were driving mammoths off a cliff and actually I think it was quite pertinent because what was going on was a whole bunch of young neanderthals learning how to drive these mammoths off a cliff with a couple of old guys stood towards the back observing what they were doing and it struck me that that is actually how an awful lot of real learning works you've got people getting stuck into real world problems learning how with each other with a bit of expertise on the side they can draw on and I think the model that you put up towards the end is far closer to the neanderthal than it is possibly to the Socratic and indeed the tools we have now may enable us to adopt an old and effective model of learning I wonder what you think of that That's interesting I hadn't thought of going back to the neanderthals but I did think of going back to maybe the late 19th century or early 20th century where we had the apprentice model being more common and so I look at Olin College and seeing that as being something like the apprentice model so they're still in a college they're not owned by a corporation but they're on task doing and learning and it seems like that's a good model and maybe we we're trying to protect our young people too much by keeping them in school, keeping them segregated from the real world too long I think especially with all the influences they have now they're ready to grow up earlier and we should have them out there if not in a formal apprentice program at least something that's approaching it where they're interacting with the real world more We've got another question there and then one at the back and then that's that'll have to be it Thanks for the refreshing lecture Peter from Universitas21 Global and I have noticed that there's a trend in which people have started you know differentiating between a search engine and a knowledge repository like Wikipedia and Wikipedia has become such a huge volume of knowledge and there's a trend in which people would like to go to Wikipedia to search out the knowledge they want rather than going to Google for searching information of any sort So I do you think that Google is facing competition in future is about to face competition from future from Wikipedia and what are the strategies which Google would like to have in order to maybe venture into the same kind of a knowledge creation format rather than knowledge searching format of business Thank you We love Wikipedia we're you know big fans of theirs and I forget the exact statistics but I think we give them something like a third of their traffic so people are going to them directly but a substantial portion of people who get to them are coming to them through us and you know I don't really see that as a threat of people going away if they know they want to go to Wikipedia I'm happy for them to go directly there but most people I think will continue to go through these more general sources because they know they don't want all their information from one place they want to have a choice and so I think people are going to continue to do that Okay we've got one last question at the back My question is a kind of follow up to that that you showed us the Google VLE at the start made a comment about that and one of the things I feel is that the Google techniques and algorithms provide us with too much information I'm wondering if you see users and user generated content as a way of providing more refined and sophisticated search techniques Okay so question is there too much information and can users help refine that so I think the problem with too much information is that there really is that much information it's not just a problem of overloading people with the presentation so we'll show you ten results and we'll say this is one to ten out of a million nobody really cares that there's a million results and they're not going to go to the end and look at that so the results are up there and really the question is how much time do they have to investigate this area how much time are they willing to put in and can we find the good stuff for them in that allocated amount of time and given that I think that there is room for user generated commentary to help that process and to an extent Google has always been driven by user generated content so there are users who happen to be webmasters who publish material there are other users who are webmasters who link between the material and that provides the sort of votes that we go on to judge appropriateness we also go by looking at our user's traffic to judge what's important and so on so it's done by algorithms but all these algorithms have as their inputs actions that ultimately come from the user now I think we can use in addition to those sources that we're already using we can use some more explicit ones of people voting yes this is a good site for this topic or for this keyword or for this area and getting more people than just the people who traditionally have had access as webmasters trying to open up that more democratize it more so that other people can put in their voices as well and so that's an area that we're certainly looking at that's all unfortunately that we've got time for I'd like to thank Peter Norvig very much for an entirely appropriate and stimulating and thought provoking talk and entirely free of powerpoint bullet points as well so thank you very much indeed for rounding off the conference in such a stimulating way