Seeing with Sound (sensory substitution for the blind)

The vOICe seeing-with-sound technology for the totally blind ( http://www.seeingwithsound.com ) featured by Rick Lockridge on TechTV in "Tech Live", reporting about blind user Pat Fletcher.

A higher quality version of the video clip is at the URL

http://www.seeingwithsound.com/media/techtv2002.asx

"It's like it has dredged up memories of sight," Fletcher said. "The sounds trigger the memory for me."

Fletcher is the No. 1 beta tester of The vOICe System. The three middle letters in vOICe stand for "Oh, I see."

Invented by Peter B. L. Meijer, The vOICe software translates images from the camera on-the-fly into closely corresponding sounds, allowing users to "see" live camera views of their surrounding environment using their ears.

The method is not based on sonar or echolocation, but instead uses real visual input from a normal webcam.

It allows blind people carrying a notebook PC equipped with a webcam to hear live views from their environment through stereo headphones. Thus, they hear the very same shapes and things that their sighted friends see.

"Over time, your mind gets used to the patterns," Fletcher said. "It likes to see."

TechTV recently visited Meijer in Eindhoven, Netherlands.

"The image gets scanned from left to right," Meijer said about his invention. "And pitch is associated with the height in the image. You hear little soundbursts corresponding to all the little bright squares."

But Meijer is quick to point out The vOICe's shortcomings - it does not convey depth perception, it can't track moving cars, and it's hard to master, he said.

However, he said he hopes smarter cameras and faster laptops will fix those problems in the future.

"I would be most satisfied if I could demonstrate that this system can provide a, let's say, level of low vision to blind people," Meijer said.

Fletcher says she believes he's already succeeded.

"Primitive though it is, it adds that extra extension that enables me to live in the world comfortably and happily," she said.

Category:

People & Blogs

Tags:

• yt:fmt=18
• Brainport Eindhoven
• sensory substitution
• seeingwithsound
• blind
• vision

[jjovereats]

@WildKatSpeaking Totally or legally?

[jjovereats]

Oh I see!

[smartsight]

@batfly That is what Paul Bach-y-Rita worked on for decades, starting in the 1960s. Nowadays his company Wicab is opting for electrodes on the tongue. Visit vision dot wicab dot com for more information about their BrainPort Vision Device.

[batfly]

@smartsight

What about combining like a grid on the skin of the back to create instantaneous little prickly sensation corresponding to pixels on a video? It would be like braille a vision?

[batfly]

@smartsight

Thanks for your response.... This idea sort of came to me out of the blue... And it makes sense to me what you are saying... though I still have some doubts in my mind... because I know the ear can distinguish plenty of notes while playing... say piano... and listening to an orchestral composition with multiple instruments... But I can totally understand the listening sense is seemingly limited when compared to vision.

[smartsight]

@batfly In vision by eyesight, the "unique pattern" of a pixel is just the position on the 2D retina, where each position has a separate neural connection to the brain: it is mostly 2D already, versus 1D for sound where two sines of the same frequency but different phase do not have separate neural connections to the brain but need to be distinguished via some other trick, such as not sounding them simultaneously.

[batfly]

@smartsight

I wonder why unique patterns don't become perceptually confusing when there are many of them in vision?

[smartsight]

@batfly Unique patterns become perceptually confusing when there are many of them. Think of having a dozen unique patterns for a dozen different pixels sharing the same tone frequency. Many image-to-sound mapping schemes work as long as there are only a few pixels of interest in the view but break down when there are many pixels, as with common environmental views, the shape of a function plot, etc. Distribution in time is then a powerful general means for disambiguating information.

[batfly]

@smartsight

Unless you give that pixel tone a unique on off pattern?

[smartsight]

@batfly A pure tone for a pixel on your left and a pure tone of the same frequency for a pixel on your right would together sound as a single pixel straight ahead. Surround sound unfortunately does not help with such ambiguities because the issue is mathematical: the sum of two sines of the same frequency becomes a single sine no matter what their respective phases are.