 First of all, let me thank you. That's a tool to interrupt, yeah. Yeah, I'm sorry. They can't interrupt until we hand that out, that's right. I'm going to confess to you that I'm having trouble speaking, so I'm going to be drinking coffee during the presentation, and at some point I'll need to get rid of it, but... I'll do the best I can for as long as I can. Let me start out by pointing a few things out to you. The audience should know. This is not my topic, it's not my area of expertise. So, at best I'm going to try to ask questions and hope that I will learn from you as opposed to the other way around. I've found that by asking naive questions sometimes you learn a lot. Now, in order to make this actually work on an interactive basis, I have to tell you I'm hearing impaired. I wear two hearing aids. I'm 65 dB down. Those of you who are in the plenary presentation know that I asked everybody to use this little microphone with an FM transmitter in it. So, when we get to a time very quickly, when this turns into a discussion, I'm going to ask that those of you who want to say something, please use the microphone and this little transmitter, otherwise I won't be able to hear your questions. And even if I couldn't answer them anyway, I would like to know what the questions were. So, let me start out by reminding you that there are a lot of very conventional mechanisms that are associated with parallelism. Mobile instruction streams with multiple data, single instruction streams with multiple data. Pipelines, and those have all been mentioned in some ways or another already. In our world at Google, map reduce turned out to be an incredibly important mechanism for making use of parallel computation in order to speed up and to reduce latency in the searching of a very large index of the World Wide Web, for example. And then there's sort of natural parallelism that happens in cryptanalysis, for example. You can hand out an encrypted string to a large number of processors and tell each one to explore by brute force a particular portion of the key space. That's a very natural decomposition into parallel operation. Or in the case of our clouds at Google, where we have a large number of users and a large number of things. Some of them are doing searches, some of them are editing documents, some of them are doing email. It doesn't matter what they're doing, what matters is it's a large number of them doing it. So we have a large number of processors that will do something for each of those parties. We distribute the computation that's required in order to satisfy each of those users. And by that means take advantage of the natural parallelism that each user brings to the problem each person wants his own particular problem to be solved. I think for some kinds of applications the biggest issue that I think about is moving intermediate results from one processor to another. A good example of this would be weather simulations where you have voxels of volume and you're trying to compute some weather components inside of each volume. As you run this computation intermediate data has to get to the adjacent voxels. And if you've lodged the voxel computation in one particular machine or a particular processor in a multi-core system you have to figure out how to move that data promptly and quickly in order to keep the computation going and that turns out to pose a fairly severe problem. And just to go back into anti-diluvian history some of you probably studied the early designs of parallel machines the ILAC-4 was one that I got introduced to fairly early in my career. This thing had an enormous number of processors 64 of them. But when you keep in mind it was built around 1974 each processor was about this big and it had a big hole in the middle it was a D-shaped piece of technology the hole in the middle was for wind to blow through to cool the thing off. It had a 1024 bit wide data bus but the thing that was the most bizarre about it is that because the processors were interconnected in a particular way in order to make use of them in parallel you had to install the information the starting state of the computation had to be put in place in a very skewed way across each of the processors so that after each cycle of computation the data that was produced in that cycle could be handed to the next processor at the appropriate time and in the appropriate physical place. So there was a huge amount of setup to take advantage of the parallelism in that machine. It also had a mean time between failure of about 30 minutes or so but Dan Slotnik, the guy that designed the machine used to say, hey, you can do a lot of computation in that 30 minutes, so you know and your point was the programming language was some very obscure thing called Glypnir, which I always thought sounded like some sort of small dinosaur it wasn't widely known. So these are all tactics which I'm sure everyone of you is very familiar with. I think the point I want to make in this slide is that problems that could be attacked in parallel don't all have the same properties and so these tactics will apply to one kind of computation but not necessarily to another so anyone who thought that you could come away from this workshop with the idea that a particular design would solve all parallelization problems should not come away with that thought you should disabuse yourself of that. Some computations and some applications will work well in some designs and not so well in others. So here's where I get into the highly naive and very speculative territory about which I know almost nothing and you can figure that much out just by reading these questions. So one speculation I have and I'd like to get your feedback on this so we have the microphone there and I'm going to turn this on okay this has side effects on my there my hearing aids have just automatically turned off this is weird okay that's on it's a green light this is the FM transmitter this is an FM receiver and there's a bluetooth system inside my hearing aids I'm a walking demonstration of the importance of batteries and high-tech okay so in theory if you want to respond you use both of these gadgets we'll both hear so with regard to multi-core chips I have the naive belief that putting more processing power on a chip doesn't do much for you unless you can get data in and out of the chip fast enough or get it back and forth between the cores and my impression right now is that has proven to be a big problem there's a hand up over there who's got the microphone you've got it okay our previous speaker so he's getting even with me for asking questions good for you just got ahold of my end of the conversation so when I first started with this stuff they weren't quite as expensive as they lacked for but these things were I remember walking across the parking lot with five CPU boards carrying three times the price of my health in my arms it was a BFM it was a big machine so you wanted to make sure you used all of what you already had today these things are really really cheap and so perhaps we go more towards all of a Swiss Army knife where you have you never use all the blades all at once maybe somebody has but I've never seen it they don't have any fingers left very good so the thing is there's a lot of special purpose things you can throw some transistors on to do and perhaps they're only used one percent of the time but if that can reduce the workload and then you don't necessarily need as much I.O that may be an alternative approach so wait let's stay with on this topic for just a second yes now the second to the last bullet I've been reading just sort of almost by accident comments about floating point not floating point field programmable gate arrays and their utility in parallel processing can you speak to that because it sounds a little bit like the Swiss Army knife observation except in this case you keep changing the blade yeah I guess I guess if you have a machine shop you can do that with a real Swiss Army knife but the place where it's been the most effective has been an embedded because you have these little mp3 players and the CPU may light up every few minutes because the rest of the time it's just having hardware accelerated pulling from memory running through some mp3 decoders and then out into the hardware and we haven't had quite that I'm not sure why but that sort of programming approach hasn't made it as much to the servers or the desktop class my guess the easy answer is well because they're general purpose they do so many things that it's harder to come up with a piece of hardware that's going to be worth its weight but they are using them in I hear that the our friends on Wall Street use it to potentially collapse our economy much faster as John Corbett put it many years ago terrific just a serious comment after this there is a talk about multi-core versus FPGAs at mid-afternoon so that you will miss yeah I'm sorry okay so let me go on with this business about moving data around in the system leads me to ask again about how we connect things together one of the things that we haven't I don't think successfully exploited yet is optical interconnects we can do it in a very crude way by connecting maybe a motherboard to a bunch of other machines using optical fiber but I wonder about free space optics there was a time when I was at DARPA where some speculation was made that we should build a set of processors that were on the inside of a sphere so that they could all see each other and the idea was that you could spread data back and forth among the various processors using free space optics and it has the nice fact it takes advantage of the property that light doesn't interfere and so you could be transmitting simultaneously without having any interference so could I ask whether there is currently known to you any activity involving use of optical communication to improve our ability to move data around in a parallel system I'm not seeing any hands going up yes, here we go again this is a real challenge I apologize for making this so hard God what broke Sorry, there's a lot of hassle for not very much comment I saw something at an HP presentation from HP Research in 2008 or 2009 experimenting with replacing the bus in a computer system with optics and they were using loops of fiber optics using interference loop to get the data but that was a more or less conventional fiber optic interconnect it wasn't free space optics what about on-chip optics as opposed to is that included in the design on the chip? no, okay so this is inter-chip or inter-board communications, okay you had a comment also I think I guess is that waveguides or fibers will be needed just because otherwise the energy density is going to be painful, you might guess I might be wrong about that okay, so we're down in a space where I have not enough intuition to have an opinion at the helm I was there was another comment I'm sorry this is making it harder for you to do but it's sure helping me I appreciate that I think with optics one of the main benefit now it's okay I can hear you but is the microphone working too? yeah, okay I think with the optics one of the main benefits is for the chip layout because you can have sort of optics going through each other whereas exactly, okay so literally you don't have metalization that's crossing which you have to separate and everything else well it would be interesting to see whether that actually gets taken advantage of if we ever go beyond silicon to things like diamond substrates and so on we may find other interesting possibilities I want to, these are not in any particular order I would guess that most of you probably have not been inside a Google data center has anybody here actually been inside of one? we have a few people from Google perhaps or spy is one of the two as a software guy I don't usually think much about the hardware that it takes to do large scale computation of course I'm always impressed when I see things like the crane machines and the connection machine and some of the other beautifully crafted things but the scale of a Google data center is utterly mind boggling some of them are 400,000 square feet they take 128 megawatts of backup power you have to cool the damn thing off I tried to get them to put pizza ovens in the top of the racks and I figured we'd get double duty out of that but they said the cheese drops down into the circle bad idea you are I was anyway completely stunned by this just physical scale of this thing and my first reaction since there are wires all over everywhere was I'm thinking my god how does anybody keep straight that the data gets from where it's supposed to go to where it is where it's supposed to go however if you were to step back and look at the way the systems are built what you see is memory and processing integrated together each circuit board has local cache has RAM it has terabytes sometimes many of disk capability on each board and of course the boards are all stacked up and there's optical interconnects everywhere running at 40 gigabits a second or more so in a very funny way it's an architecture which is mixing together memory and processing adjacent to each other so one of the questions I wonder about is whether we should be exploring more ways on a smaller scale of joining processing and memory together in a sense taking advantage in the same way we do in the Google data centers but on a smaller chip size scale there's a coming this is one of the places where FPGAs do take the win because they have this distributed memory so you have computation at a bit level right next to each other distributed spatially in a massive array across the chip so if you can make use of that through physical structure of the computation then that's why for the appropriate design an FPGA will wipe the floor every day with a CPU and that's one of the elements it happens at a bit level rather than at a CPU gigabyte memory terabyte hard drive got it thank you for that oh yes please go ahead yeah one of the ways where it is actually done on the CPU level is when the Erlang guy started to rewrite the Erlang engine to run on multiple CPUs they try to come up with a scheduler that makes sure that an Erlang process is always staying on the same CPU having the data in the same CPU cache without the need to actually shift data between CPUs and so on so that you get around those limitations so just on that score I did have a small exposure to one of the scheduling problems that we have in the data center and that's figuring out for this particular query or this particular email interaction which machine should this run on and clearly when you have a fairly large chunk of data that the user is working with like maybe your Gmail it's important that that particular interaction occur on a place where the data is already located so there were a number of algorithms that were explored experimentally for scheduling and distribution of computers to the application and I was quite surprised to learn that first fit turned out to be the most effective algorithm it's the simplest one you think of and you think well that can't be the right answer because it's too easy and there is such a thing as being overly sophisticated so that's how we ended up with our scheduling algorithm yes sir? Are you aware of any algorithm that has been written with the purpose of saving power? Oh that's interesting the answer is yes but not in the context that you intended the question there have been a number of algorithms related to mobiles for example to try to reduce power for transmission in order to save battery power but I love the question because the idea of dealing dealing with computational power consumption is for us a big deal anything that would reduce the amount of power required to run a data center would be of great value to us and I don't know that we've actually tried to do that there's a hand way in the back we're going to have to throw the whole shebang up there Well while I go there I just tell the story that it's not Mike from Beckman who is a sort of the argon something bad in Chicago and they have this computational problem that they just need to switch off the machines in summer Yeah I just wanted to say this isn't my research but there's definitely a research group that's working on this kind of stuff they've got heavily instrumented machines and they'll run multiple implementations of a problem and they can tell you to the milli watt how much the CPU drew during the calculation and I'm sure there's other research groups doing this stuff too I know that there are more gross efforts at reducing power requirements by turning various parts of a chip or parts of a circuit board off that appear not to be needed but the big issue there is what happens when you need them, how long does it take to turn them back on Somewhere in here I had, yes memory serious I wanted to mention this now this is a technology about which I do not know much but my impression is like the FPGAs they have the possibility of simultaneously using the same basic physics to both remember something and also to compute with it the consequence of that is exactly the point about coalescing the computing power and memory and the nice thing about the memrister is that it doesn't require any power to remember the information that it has in it and so you have effects like instant on and a few other things that might contribute to power reduction because you don't need power in order to maintain the information in the system Yes where is the next one I just wanted to say that our group is working on power management and power reduction within multi-core and one promising approach is heterogeneous multi-core systems where you've got low powered processors and high powered processors you turn all the high powered processors off when you don't need them and when the low powered processors start overloading you then fire up the other one and migrate the processors and we're exploring that with the panda board at the moment which has got A9 and M3 processors all in the same chip and so missing so far So now an interesting question is whether the FPGA could be used in a way that would allow it to configure itself to be the high power or the low power thing depending on what you needed or is it the basic implementation that's going to consume whatever power and that's it I can't speak to FPGAs but I do know that manufacturers are now putting low powered and high powered cores on the same substrate on the same chip If you don't mind this ancient dinosaur going back into ancient history again my thesis advisor at UCLA was a man named Gerald Estrin and in the 1950s a really long time ago he started working on something called a fixed plus variable processor they had a digital processor and they had an analog processor and the analog processor ran faster than the digital processor in terms of getting solutions except for the fact that it didn't get very many digits of accuracy so the idea was to use the variable function to get a starting point to feed the digital now this is 1955 or 56 what occurs to me is there's some similarity to this idea of having higher power and low power or mixed heterogeneous elements in a common chipset are your results being published? Yes they will be great are there other comments or would keep going? now some of you, if you're like me a big science fiction fan and I love to talk about or at least read about quantum communications and teleportation and all this other stuff but there is something going on in entanglement which continues to be very puzzling we all know about this spooky interaction at a distance and Einstein didn't like it at all my impression is that we keep hearing more and more ways in which entangled particles or entangled elements can influence each other regardless of how far apart they are and this makes me wonder whether there's something going on in the physics down at the quantum level that we could take advantage of somehow if it's true if you look at string theory for example the physics through the string theoretic view you get something like 11 dimensions and a lot of symmetry it's conceivable that some of these effects which are demonstrable are taking place through dimensions which are not the four that we're familiar with the spacetime notion that Einstein used for his theory they say well we can't see those dimensions because they're all very tiny and they're all wrapped up but it would be interesting if it were actually the case that these entanglement effects are propagated not through the normal spacetime dimensions but through these other dimensions if they exist if we could possibly take advantage of that we could conceivably allow communication over substantial physical distances so we could build a larger and larger scale computing engine which doesn't suffer from speed of light delay and any of you who have designed machines that are up against the propagation speed of electricity or electrons in a wire of photons in an optical fiber will know that that's a severe limitation and if we can get past that it would make a big difference yes sir as for faster than light it's not current state of physics is that it's not possible we do have systems where you can split two entangled bits and collapse one of them and the other one is guaranteed to collapse in the opposite direction but you can't actually get the information out until you compare the two ends via a normal channel notice you're wearing an interesting equation there which one is it let me only arm wrestle with you a little bit because I actually understand and have heard the same thing the only reason that I raise this is a not so totally crazy idea is that there's still some peculiar evidence in the theory behind quantum computing that the reason there's a quantum speed up is that some of the results are coming from the future in other words this notion of present past and future remember I'm not a physicist so I'm struggling with this but the idea that all three of these states are interacting with each other and that some information is actually apparently coming from the future suggests that we may actually get to the point where we can force state your point is you can't force the state of an entangled particle in order to cause the other guy to take on a certain value if we could get there of course then we would have faster than light communication there still is a problem you have to create the entanglement you have to move half of the entangled particles to the other place physically before you can take advantage of that so you still have a speed of light delay in order to get the system set up I agree with you we don't have strong evidence at this point but I've seen some really surprising results where it's been possible to teleport that is to say reproduce the state of a particular particle by using three of them in a series of transactions in effect it causes a particular electron to be propagated from here to there what's really happened is its state has been propagated from here to there remember all electrons are indistinguishable except for their spin state so teleportation really means recreation of this electron in some other place with the same state I'm waving my hands like crazy here but it's fun to imagine that we might actually to build something like that we can teleport in the sense that we can recreate the state as we're doing that faster than light that's an issue surely you're going to solve that problem for us somebody has to do that okay yeah so we'll get you next no no wait I have him back here and then we'll get it just out of interest there is actually a patent granted by the Australian Patent Office on superluminal communication did this go to Buck Rogers or something? no I was a guy who wears tinfoil hats and yeah crazy dude rocked up and gave a presentation at a group I'm part of the evidence for superluminal communication being possible is that he has a patent on it oh I see it must be a certain circularity there right okay we have one down here yes sir well this is another humorous circular comment Google could end up using data centers in the future before they built them I wouldn't be really my head hurts I'm waiting for the answer well you know if we could figure out how to do that and get somebody else to pay for them that would be wonderful it would do wonders for our business model yes sir not sure that I understand all the physics behind I know neither so don't worry about it taking having two particles and having to collapse one and as you said the speed of light means you can only move that particle you just start a stream of particles off and they're received at the receiving end stored and when you need to send information you collapse the ones you've sent earlier that's the theory the problem of course is that you do have a speed of light limitation if you physically have to move the particles to the place where they're going to get used I've been dying to find some solution to the FTL problem because every time I look at inter interplanetary communication which is already hard enough I'm now starting to work on an interstellar mission and part of the problem is that at current speeds we only deliver an object to another star even a nearby one in about 50,000 years it's kind of a long time that considering you know human society hasn't existed more than the fifth of that so the next question is whether we can get anything that would go faster than that and can we deliver them in like 100 years or 40 years or something even to the nearest star like Alpha Centauri but if we could deliver entangled particles in let's say a 40 year period at a tenth the speed of light then we could use this kind of if we have the ability to actually force the state of those particles we could use this quantum communication to make it work that's all kind of speculative living high speed how is your interplanetary corporation doing these days well actually we are well along in establishing standards for interplanetary communication I think I mentioned in my plenary remarks the consultative committee of space data systems which standardizes communication for spacecraft is already considering the adoption and use of these interplanetary protocols for standards purposes and if we get that all done by the end of let's say 2011 we have a shot at having at least the wherewithal to expect interoperability among all the various spacecraft that are being built and that's a very exciting prospect because we can repurpose them to be part of a network so I see a question over there yes sir with the entangled particles now I'm not a physicist but I do know information theory and just the fact that the entangled particles has collapse is itself information but if you've got two particles and you can collapse one and then the other in a particular time interval well that is information you may not have to worry about the state oh great we're going to have a fight by the way before you answer I have to tell you in this conversation how many times have you heard the I am not a lawyer now I am not a physicist is the new thing I actually am a physicist when you read the particle like when you find out it's that you can't tell whether the other end has collapsed it or not you only see spin up spin down so I look at the particle I see it spin up and once I've read that then the other I know that the other electron must be spin down but I don't know if they've collapsed it yet or not no you know I don't know what's the matter with you physicists you ought to be able to make this stuff work haven't we done enough yeah okay do you have other slide apart of the physics one I have some other slides I'm going to ask about graphics processing units some people apparently have used GPUs as a way of because they're very super speed they do all this rendering very very rapidly some people have been able to literally subvert the purpose of the GPU into doing computations you have to do crazy things like expressing the problem in very peculiar forms like which color is this answer I haven't ever I haven't ever actually jumped into the middle of all that I don't know all the weirdness but I thought maybe some of you may have had the experience of taking GPUs and turning them into a multi-processing system and you could tell us about that so I see the physicists over there you're not go for it you don't actually have to subvert them anymore all of the state-of-the-art GPUs are general purpose single instruction multi-data machines that just happen to have firmware seen off to the side to do all of the various three-space trig that you need to do it's very easy to play with now you can go to Amazon Web Services and for $16.80 US you can buy 1 trillion floating point operations on GPU I have been known to pay more for a pot of tea wow okay there's one way back there this is starting to sound really cool have you actually done that and what kind of computation were you doing I'm sorry maybe you can't tell us oh I'm sorry I can't hear a thing because he's got the microphone okay we have here yes so as the speaker was saying yes there is lots open CL there's CUDA languages for talking these things and they're getting at the moment still IO as soon as you've got to go and try and fetch data that's not actually stored on the GPU I'm reading a paper one of the guys was doing make sure you had all that data as close as you can get it because general purpose CPUs are still doing better in the case where they've got to go and fetch from main memory they're not designed to do that efficiently okay thank you one more comment from our physicist here you're now on the title resident physicist I think is anyone else before that you know Mr Chairman I apologize for making you run all over the building here I told you I would have fun this is the last time you'll ever have a deaf guy come and talk the point of the whole mini conf is not to think what we have been doing because there is a lot of history here but start to think about what can be done so off to the physicist answer is not actually going to be physics related one of the other problems that you have with GPUs is that they're only really good for highly homogeneous problems because every core has to take the same branches and the way they do it is just by turning off the cores if they're currently executing a branch which they're not supposed to be in okay so that means the set of problems to which you can apply this is going to be constrained by that this is like the hot dog that didn't make it already my name is Matt I do weather simulation aerodynamic simulation for foreign government using classes of GPU devices and the real challenge we work with numerous algorithms and the real problem is changing the original algorithm to operate efficiently on a GPU device so you can't just take any algorithm and throw it on it's just not going to work so you have to take them to account the hardware when you design the algorithm so using that we're able to get performance 2,000, 2,500 times faster than on a normal CPU cluster and the foreign government I work for is not that big we don't have a lot of money so that's why we use it the rest of us are going to be very curious to find out which foreign government you're working for and what are they doing with all that processing power just a comment about what we had on the previous mini-conf about heterogeneous computing about we have all these chips but have different colors of chips so that was a question that we discussed on the previous mini-conf a year ago and everyone is welcome to comment on that if I could just reinforce the comment that you made about the need to adapt the algorithm to the hardware there's no question in my mind a lot of the successes in parallelism have been rooted exactly in understanding the computational nature of the underlying hardware and the ability to rethink how to get to a particular computational result and this earlier comments that you made by the way about whether it's a synchrony or a commutativity which we didn't get to talk too much about or the possibility of approaching a computation through different paths different sequences is really important because if you can prove or at least persuade yourself that the computation produces the same result even if it goes through a different sequence of steps then you may actually be able to adapt to these special pieces of hardware who's got the, there you are no, no, who has it, now you do okay go getting back to the types of jobs that you do the AWS, there's a blog post somebody's done a proof of concept of using an AWS compute node with the Nvidia Fermi on it to brute force is it SHA1 or SHA256 passwords and they've actually got a dollar cost for doing that so if you have a budget you can budget, you know, I have a grand to spend on cracking these passwords how many can I do? so the question now will measure in cups of tea I might just wave my FPGA flag again briefly one of the comments about the issues with GPUs is well we've got to structure our algorithms to fit these architectures and with lots of conditions aside again that's one of the promises of custom computing machines is well you structure your architecture to fit the computation so you turn it the other way around and so some really simple examples of this which also come back to the power consumption question that was raised before is when you have this flexible computing fabric you can make a fast and large and high power architecture to solve a computation or you can make a small slow low power architecture to solve the same computation and there's this sort of this rule of thumb in the research area which is basically that the time area product is roughly a constant so you can just kind of pick a point on that on that with a lot of conditions as well but anyway there we are noticing that we're going to run out of time in a little while so so I had some additional hacks I want to talk about anyway go ahead I suppose my general thought on this I had been using now I can't remember the name of the library for doing parallel computation for a number of years and I had read a paper on how to do parallel sorting which I always thought would be something that would be incredibly linear and in fact it works quite well what they used to use on data tapes and thinking of now I can't remember the name of the guy who runs e-software but John Carmack his comment was the hardest challenge that games designers now have to face is multi-core systems they're used to having to do everything but they have all of the resources of one CPU now splitting that task up is actually the hard part and I suspect we're going to see for specialized workloads the effort goes into making it easy to parallelize those and we're seeing all of the work going into that but for our general workloads people are still solving these things the wrong way because they've got the wrong tools at the software level and it would be nice to have that whole layer there and I think it's going to be all of those building blocks that are going to get there rather than just get the hardware right or get the firmware right this is in some ways this is not any different than getting an automobile engine to work it's not a question of just putting gas in the gas tank there are a whole bunch of other parameters that you have to make sure are properly configured and works or otherwise it doesn't I agree with that wouldn't it be time to have different operating system for each of these areas I mean the person I've been talking about whether prediction that or in order to simplify I don't know whether I would be persuaded that you need a different operating system for that exactly one extremely nimble straight to bare metal okay well let me ask we've already talked about the play station related stuff the GPU stuff but what about things like the search for extraterrestrial intelligence the SETI at home by the way just a side comment you know why we're doing this search for extraterrestrial intelligence because we didn't find any here so we're looking at the one thing that's interesting about that proposition and the substrate that it runs on is that it is taking the problem and distributing it to a very large number of potential players and it works partly because it's naturally a parallel reception of a large amount of frequencies and a large number of processors that are waiting and will accept a certain amount of data so it's a confined search and pattern detection problem the same is true for protein folding so that's a kind of parallelism that doesn't necessarily require a very very rapid interaction because the problem is patient in some sense we're just doing pattern recognition we don't have to do it instantaneously you've probably seen a similar way of using people as parts of a parallel processing system we do a lot of work at Google in scanning of documents for the you know this sort of thing for Google book search and often our algorithms don't recognize some of the characters you'll know that in some websites you don't get to put anything into it unless you type a rather obscured image of some words so some people have put the deliberately obscured image plus the thing that we couldn't recognize and as a user you'd have to do both and we figure if you get the deliberately obscured capture correct that you probably got the other string correct and we use that to improve our character recognition so we're using human beings as part of a parallel distributed processing system nobody takes that the wrong way I mean it sounds kind of like the matrix with everybody all plugged in okay other comments way in the back on my way we gotta find a better way to do this it's really hard for everybody on the note of people to decode something you don't recognize there is a group of people out there on the internet who have realized you're doing that and are actually all using a rude word for that second word so they're trying to get the wrong words recognized okay next comment can we go back to the physicists now can I stop with the synthesis I was just wondering what the energy efficiency was of human beings doing the captures god I don't think we ever measured that that's a good question but since we don't have to pay for that part that isn't a big issue okay I actually have to kind of wrap up now because I have things I need to do before the end of the day what else is funnier than this first of all thank you very much for telling me teaching me some of the things that you know that I don't I hope I get to come back and learn some more from you in another time in the meantime I appreciate the excuse to be here in Australia and I want to say that the people of Brisbane and the folks who organized this conference have really impressed me with their can-do ability I don't think I've ever seen anything like this so to be very congratulated I hope others can emulate that too thank you very much and please join me