Loading...

SUGILITE: Creating Multimodal Smartphone Automation by Demonstration

97 views

Loading...

Loading...

Transcript

The interactive transcript could not be loaded.

Loading...

Loading...

Rating is available when the video has been rented.
This feature is not available right now. Please try again later.
Published on Jan 4, 2017

SUGILITE is a new programming-by-demonstration (PBD) system that enables users to create automation on smartphones. SUGILITE uses Android’s accessibility API to support automating arbitrary tasks in any Android app (or even across multiple apps). When the user gives verbal commands that SUGILITE does not know how to execute, the user can demonstrate by directly manipulating the regu-lar apps’ user interface. By leveraging the verbal instruc-tions, the demonstrated procedures, and the apps’ UI hierar-chy structures, SUGILITE can automatically generalize the script from the recorded actions, so SUGILITE learns how to perform tasks with different variations and parameters from a single demonstration. Extensive error handling and con-text checking support forking the script when new situa-tions are encountered, and provide robustness if the apps change their user interface. Our lab study suggests that us-ers with little or no programming knowledge can success-fully automate smartphone tasks using SUGILITE.

Loading...

When autoplay is enabled, a suggested video will automatically play next.

Up next

to add this to Watch Later

Add to

Loading playlists...