Sensory substitution for the blind: a walk in the garden wearing The vOICe

The vOICe gives blind people live detailed visual information about their environment. In this clip, use is made of camera glasses and an optional wide-angle (fish-eye) lens. More information about this wearable setup is available at http://www.seeingwithsound.com/mini_d...

Normally, knowledge of context helps greatly with visual interpretation, especially in familiar environments. Here we compensate with a coarse annotation by time point for blind people who watch this video clip (ideally a screen reader would access and speak the closed captions):

0: I walk across the terrace towards our neighbor's house, with our own house of white bricks on the left side. Bright sky is at the top right.

10: a white garden chair shows briefly at the bottom left, and the tonality of the bright cement of the brick wall of our neighbor's house becomes apparent as I walk towards it until it is within arm's reach. Then I look left and right along the brick wall such that you get the changing rhythm rates associated with visual perspective.

37: then I take a closer look at the white garden chair, and pass it to walk towards a wooden fence, with a white football lying on the ground in front of it. The football gives a low-pitched beep while the wooden fence gives a slowing rhythm as I move closer.

1.08: I turn left to face the wall with white bricks of our own house, giving a more noisy texture with a tonality from the darker cement, and I move a bit closer to the wall.

1.25: I walk around a kind of white brick pillar at the corner of our house, and approach it from the side that has a rain pipe in the middle, and move up close and put my hand on the rain pipe.

1.55: I turn a bit left and take a close-up look at a red table cloth with white dots on a garden table. The many white dots give a peculiar sound texture.

2.12 I turn further left to take a closer look at a corner of our house, with windows on the left and right right side of the corner.

2.22: I turn left and walk over the grass to another wooden fence, giving a characteristic rhythm that slows down as I move closer.

2.33: Turning a bit to the right I take a look at some nearby foliage giving a kind of random texture.

2.45: Next I take a look at the roof with its regular pattern made up of the rows and columns of stone roof tiles.

3.06: I walk across the terrace to a white door in the fence that faces the street behind it.

3.31: I briefly turn right looking towards the white brick wall, and then turn left for a close-up look at some other foliage, and turn further left to effectively make a U-turn after facing another brick wall.

3.58: Then walking back across the terrace I turn to the right to have a close-up look at an empty rabbit's cage, which here gives a characteristic simultaneous tonality and rhythm at close range, corresponding to a rectangular metal wire pattern.

4.08: I walk further across the terrace to take a second look at the white brick pillar with the rain pipe, again putting my hand on the rain pipe at close range.

4.38: end of video clip.

After extensive use, brain plasticity should help make use of The vOICe "second nature", with less focus on the sounds and more focus on the visual content carried by the sounds.

• Category

  Science & Technology

[radya a,k]

My Ears is hurts. but it works like an angel! . im not blind. :D

[AppA]

This sounds magnificent.

[Oriru Bastard]

Yeah no... I'd rather hit my head to a wall than hear than sound constantly.

[OpticalFascism]

Poor blind people, next up they'll become deaf :(

[Dzung Nguyen]

Well, this is ambitious. But i have to say that the technology is so simple to be a useful device. Furthermore, the constant wave sound is not really pleasant to the ear ..

[Heligoland43]

Oh wow. Go to the site in the bar, and head to the main page and find the demo button on the bottom. Now just start to make your own picture. draw like a straight line, horizontal or vertical, and see what kind of sound it makes. it will make so much more sense.

[Heligoland43]

Takes a lot of time to adjust, eventually synesthesia happens. Your brain rewires it into a sort of vision proxy. I imagine touching things, and translation from a friend is very helpful, to "learn the language"

Read the summary up top. Certain things are consistent when you look at them- so you would be able to look at that red and polkadotted table and recognize that pattern it makes.

also it helps to have the rules: louder=brighter, higher pitch=higher in the visual field

[Rafael Pinto]

I could play tic-tac-toe (the one that comes with the software) using this (I'm not blind) after a few minutes.

But my question is: will it produce visual sensations in blind people like the tong devices do?

[TiagoTiagoT]

with practice you start to understand the sound, i guess it's easier for some people than for others

[yadishansar]

I  am not blind. I am unable to match the voice with the video. How a blind person will recongnize things in detail with this strange sound?