Reconstructions from brain activity: 3 subjects

For an explanation of the procedure please see the other video on this channel: http://www.youtube.com/watch?v=nsjDnYxJ0bo
This video is organized as follows: the movie that each subject viewed while in the magnet is shown at upper left. Reconstructions for three subjects are shown in the three rows at bottom. All these reconstructions were obtained using only each subject's brain activity and a library of 18 million seconds of random YouTube video that did not include the movies used as stimuli. (In brief, the algorithm processes each of the 18 million clips through the brain model, and identifies the clips that would have produced brain activity as similar to the measured activity as possible. The clips used to fit the model, the clips used to test the model and the clips used to reconstruct the stimulus were entirely separate.) The reconstruction at far left is the Average High Posterior (AHP). The reconstruction in the second column is the Maximum a Posteriori (MAP). The other columns represent less likely reconstructions. The AHP is obtained by simply averaging over the 100 most likely movies in the reconstruction library. These reconstructions show that the process is very consistent, though the quality of the reconstructions does depend somewhat on the quality of brain activity data recorded from each subject.
For the paper (Nishimoto et al., 2011, Current Biology) go to: http://dx.doi.org/10.1016/j.cub.2011.08.031
You can find more information about this work at our laboratory web site: http://gallantlab.org

Category:

Science & Technology

Tags:

• neuroscience
• brain
• decoding
• jack
• gallant
• shinji
• nishimoto
• fmri
• berkeley

[hidenorivideo]

This is incredible, but I'm not sure the videos were added together the best way in the reconstruction (though I don't know any better way). The borders of things lose their meaningful value when 2D averaged out pixel by pixel. Are there ways to composite images based on borders/fields rather than grids?

[gallantlabucb]

@hidenorivideo You are correct, we did not do optimal averaging here. We simply summed across the pixels, whereas a better averaging technique would involve aligning the objects in the videos first. We didn't do that here because our main goal was to build good ENcoding models, and the DEcoding results that you see here are just a side issue.

[tucense]

You should really try this with someone who is sleeping and see what happens.

[SoloAWeek]

Thumbs up if you saw Hank Green in there!

[thrillzonevideos]

@dapenguin64 Now we can get sued for copyright infringement for simply having a song stuck in our heads.

[TheKrabDoctor]

0:07 top left: Earthbound lol

[dapenguin64]

Can this concept be applied to sound?

Thinking music, then sampling it?

[JLCL01]

0:27 *uppermost right* Initial D <3

Also: CONFOUND THIS VIDEO, IT DRIVES ME TO WASTE TIME!

[shiptech2k9]

lol I saw Hitler

[Glucoperon]

wow. can be used as a really good lye detector

[UnlawfulCharmeleon]

Liam Dryden at 0:28

[enriqueiglesiaslooov]

Thumbs up if Hank Green on Sci Show sent you here :)

[MrWilc0x]

Hank Green makes an appearance at 0:21, then again at 0:26.