ShopMobile: Mobile Eyes-Free Shopping System on Google Nexus One

A lab demo of ShopMobile, our mobile accessible
shopping system that runs on the Google Android Nexus One smartphone.
ShopMobile grew out of our research on ShopTalk (http://www.youtube.com/watch?v=CLIXhNjRal0), a wearable system for accessible blind shopping. The only sensor this system uses is the phone camera.

This is an experimental setup in our laboratory where we conduct our
experiments. Many stores have two types of barcodes: shelf barcodes and
package barcodes. Most stores use the MSI symbology for shelf barcodes
and the UPC symbology for package barcodes.

The first component of the system is the eyes-free five gesture touch
screen interface that allows the user to browse the shopping list on the
phone. The five gestures supported by this UI is up, down, left, right,
and tap.

The user can find the shelf barcodes on shelves by hand. Once a shelf barcode is identified, the user can tap the screen and put the system into
the camera alignment loop to align the camera with the barcode surface in the pitch and yaw planes. When the camera is misaligned the user receives vibrations of varying degrees.

The database of products used by the system is called the barcode connectivity matrix (BCM). You can check the links at the end of this description for technical papers where the BCM is described.

The system can handle 90 degree rotations. The system is fairly
resistant to false positives. If the user takes a picture of a surface with no barcode, there is a good chance that no barcode will be detected.

The system can detect barcodes on bottles and cans.

The current version of the system is based on taking still images. The
next version that we are currently working on will use automatic frame
grabbing techniques. We expect that it will make the overall experience
of working with the system more pleasant to the user in that the user
will not have to take images constantly.

The source code for this version is available for download at the following URL:
http://digital.cs.usu.edu/~vkulyukin/vkweb/software/shopmobile2_demo_june2010...

For more information go to
http://digital.cs.usu.edu/~vkulyukin/vkweb/research/eyesfree_barcode_scanning...

Video Narration: Vladimir Kulyukin

Category:

Science & Technology

Tags:

• ShopMobile
• assistive technology
• rehabilitation engineering
• accessible shopping
• orientation and mobility
• accessibility
• android
• Google Nexus One
• barcode localization
• barcode detection
• computer vision
• visually impaired
• aliasgar kutiyanawala
• vladimir kulyukin
• nexus one
• eyes-free shopping