Reconstruction from brain activity

** PLEASE READ THE CAPTION TO UNDERSTAND THIS VIDEO **
The left clip is a segment of a Hollywood movie trailer that the subject viewed while in the magnet. The right clip shows the reconstruction of this segment from brain activity measured using fMRI. The procedure is as follows:
[1] Record brain activity while the subject watches several hours of movie trailers.
[2] Build dictionaries (i.e., regression models) that translate between the shapes, edges and motion in the movies and measured brain activity. A separate dictionary is constructed for each of several thousand points at which brain activity was measured.
(For experts: The real advance of this study was the construction of a movie-to-brain activity encoding model that accurately predicts brain activity evoked by arbitrary novel movies.)
[3] Record brain activity to a new set of movie trailers that will be used to test the quality of the dictionaries and reconstructions.
[4] Build a random library of ~18,000,000 seconds (5000 hours) of video downloaded at random from YouTube. (Note these videos have no overlap with the movies that subjects saw in the magnet). Put each of these clips through the dictionaries to generate predictions of brain activity. Select the 100 clips whose predicted activity is most similar to the observed brain activity. Average these clips together. This is the reconstruction.

For a related video see: http://www.youtube.com/watch?v=KMA23JJ1M1o
For the paper (Nishimoto et al., 2011, Current Biology) go to:http://dx.doi.org/10.1016/j.cub.2011.08.031
For more information about this work, please check our lab web site:http://gallantlab.org

Category:

Science & Technology

Tags:

• Jack Gallant
• Shinji Nishimoto
• neuroscience
• brain
• decoding
• fmri
• berkeley

[TobWaylan]

Thanks, Hank Green.

[Lbrtyordth8]

@skarfaced That is a disturbingly naive comment. Who's in charge of what actions are considered crimes, afterall? It'll start with actions "for our own protection" and creep slowly on from there until we're all in danger of convictions for thought crimes.

[BlockisticStudios]

@Lbrtyordth8

the new SOPA

lmao

[tyler62126]

@kinflyer it's not

[ALeafMeBe]

it's like their brains were trying to match the image up with something else from memory...

[kinflyer]

Could anybody tell me this is real time detection or not?

[jsymons1985]

Amazing application of Bayes rule. Just terrific.

[computergeekman12]

Can this reconstruct sound? It would be cool if it could.

[alexchris32]

try this on me if you want really strange porn

[Antonis32d]

the question is... how does our mind reconstruct the videos on the right that are reconstructions of the videos on the left. brainception

[iammiahblaze]

i bet they could learn secrets of the universe by studying mine!

[RANDYRodentFanClub]

we are one more step closer to mind reading, congratulations for the study