RESONANT VISION

An fMRI scan of a subject looking at a simple visual stimulus appears here as music merged with a whole-brain visualization. Regions that are active get brighter, and their melody rises, as brain activity increases. At first, the video displays the visual cortex, which is nicely coupled to the checkerboard stimulus the subject was watching. But as more regions of the brain are added to the display, the visual activity is finally heard in a rich context of all-over brain activity.
NOTE: Since this video was first posted, the brainmusic project has attracted the interest of the independent documentary film maker Elisa da Prato. To check the status of the feature-length film (and to support the project), see http://musicofthehemispheres.com/
The "mind as music" hypothesis finds empirical support in this open source paper:
http://www.frontiersin.org/theoretical_and_philosophical_psychology/10.3389/f...

Category:

Science & Technology

Tags:

• fMRI
• brain
• animation
• music
• brainmusic
• sonification
• neuroscience
• scanning
• electronic
• synthesizer
• MIDI
• visualization
• cognitive
• perception
• vision
• visual
• system

[jestertru]

How much of the periodicity of these signals is an artifact of the heart pumping blood (and hence a kick of metabolic activity) and from the scan rate of the data extractions itself. The modulation of the sound seems a bit too close to either rates.

Also makes me think of Jazz and wonder if some of like it is that it "maps" well to how the brain's distributed processing works.

[dlloyd1984]

Glad you asked, as this will provide a footnote to the video. You're right, the basic rhythm is that of the image acquisition itself, 1 image per second. (So, each run in the video has been speeded up considerably, an aesthetic choice.) In real time, each "take" in the video would run about 3 minutes. (more...)

[dlloyd1984]

(part 2:) The fast moving components, rendered as steel drums and talko drums, oscillate about every two seconds, so these are candidates for a physiological rhythm. In principle, ICA should isolate one or two regions that follow this rhythm; these might rightly be regarded as artifacts.

The general rise and fall (in pitch) of the other regions arguably reflects the blood oxygen level and hence the underlying neural activity. These interweaving melodic lines thus may have cognitive meaning.

[dlloyd1984]

(part 3) Is it appropriate to invoke jazz? That would be nice. A musician/composer who has heard some of the other videos on this channel says that the braintracks are in 2/4 time. Jazziness might be a property of more complex rhythms; to my ear, this video does have a little swing.

[dlloyd1984]

(part 4!) In one respect, however, the jazz sound is my addition, and not intrinsic to the data: I assigned the intensity values in the fMRI signals to a blues scale. I could have used a chromatic scale, pentatonic, or arbitrary frequencies. That was an aesthetic choice. However, the mapping of rising tones to increasing fMRI signal intensity is a genuine feature of the data. That would be audible regardless of all the other choices. (end of reply, at last....)

[bsodfc]

This is amazing!

[TheNothingRoom]

I don't think you're hearing the brain's signals actually create anything reminiscent of jazz here. The mapping is to a jazz scale; you could also map this to other scales from other cultures and have a different but likely equally interesting result. In algorithmic composition the mapping typically indicates the genre, whereas the algorithmic material dictates the inspiration. Very interesting nonetheless.

[kzatching]

awesome!