Multimodal mobile interaction - blending speech and GUI input - iphone demo

A siri like (personal assistant) interface developed as part of my PhD research (focus on mutlimodal interaction), circa 2009

Multimodal interfaces (interfaces that support more than 1 interaction modalities) offer a richer user experience; they are more flexible and robust (at the cost of greater design and implementation complexity).

This video is about a multimodal mobile interaction application demonstrating how to exploit speech and GUI (touch) modalities to enrich user experience. The application scenario is a travel reservation service. The user can use either GUI or speech input at each interaction turn, that is, selecting values from a list by touch or directly speaking, e.g. "I want to fly from Orlando to Chicago next Friday evening".

This specific demonstration showcases 4 different interactions modes, one unimodal (GUI only input) and 3 different multimodal ones:
-"Click-to-Talk": user clicks speech button to talk
-"Open-Mike": speech input using voice activity detection
-"Modality-selection": default input modality chosen on modality efficiency; the system switches between "Click-to-Talk" & "Open-Mike" depending on current context

Note that the same (and also the simpest possible, e.g. one way trip without car/hotel reservation) scenario (New-York to Chicago, etc.) is demonstrated for all different interaction modes (Of course everything you can do with GUI you can do with speech). This video was shot to showcase the porting to iphone platform (with the help of V Kouloumenta); the platform also runs on PCs and various PDAs (e.g. Zaurus), since 2006.

This demo is part of my PhD work at Electronics & Computer Engineering Dept, Technical University Crete under the supervision of A. Potamianos. For more info you may refer to:
M. Perakakis and A. Potamianos. A study in efficiency and modality usage in multimodal form filling systems. IEEE Transactions on Audio, Speech and Language Processing, 2008.

Links:
http://perak.wordpress.com/2010/10/15/multimodal-mobile-interaction-blending-...
http://gr.linkedin.com/in/manolisperakakis

Category:

Science & Technology

Tags:

• siri

[norad73]

Ωραίος! Και παίζει και με ελληνική προφορά :)