Realtime Performance-Based Facial Animation (SIGGRAPH 2011)

SIGGRAPH 2011 Paper Video: This paper presents a system for performance-based character animation that enables any user to control the facial expressions of a digital avatar in realtime. The user is recorded in a natural environment using a non-intrusive, commercially available 3D sensor. The simplicity of this acquisition device comes at the cost of high noise levels in the acquired data. To effectively map low-quality 2D images and 3D depth maps to realistic facial expressions, we introduce a novel face tracking algorithm that combines geometry and texture registration with pre-recorded animation priors in a single optimization. Formulated as a maximum a posteriori estimation in a reduced parameter space, our method implicitly exploits temporal coherence to stabilize the tracking. We demonstrate that accurate 3D facial dynamics can be reconstructed in realtime without the use of face markers, intrusive lighting, or complex scanning hardware. This makes our system easy to deploy and facilitates a range of new applications, e.g. in digital gameplay or social interactions.

Category:

Science & Technology

Tags:

• Hao Li
• Facial Animation
• Kinect
• Example-Based
• Facial Tracking
• Retargeting
• Automatic
• Expression Transfer
• Animation Prior
• FACS
• Facial Action Coding System
• 3D Scanning
• Art-Directable
• Blendshapes
• Computer Graphics
• Computer Animation
• Lee Perry-Smith
• Siggraph
• Statistical Prior
• Non-Rigid Tracking
• Non-Rigid Registration

[nvrana]

Did you guys use the Open Kinect SDK?

[haoli81]

@nvrana The current version is with both OpenNI, before it was libfreenect, I believe this is the OpenKinect SDK, so you can use both

[EnneagramVIdeo]

What is your contact information? Thank you.

[haoli81]

@EnneagramVIdeo hao@hao-li.com

[ohmss006]

Hi, I would like to ask (again), does anyone know what is happening with this at the moment??? Is there an application or something?



[haoli81]

@ohmss006 Yeah, we are going to commercialize an independent software and also Maya plugin very soon. We will showcase the product at the Emerging technologies at Siggraph Asia 2011.

[ohmss006]

@haoli81 Hi agian, so sorry for the very late reply, but how is it going? and how did it go at Siggraph Asia 2011? :D

Also, I was wondering, in theory, could one potentially capture the facial animation using this method and apply it to any 3d head model??? because I might want to do some tests.

[m3htrix]

While not perfect, these are by far the most impressive results I have seen using a commercially available camera. Keep up the good work!

[justicefadingtoblack]

wondeful reseach, loved going through your article. and the results are amazing. looking forward to see it integrated in a commercial application.

[nvrana]

@haoli81 We're building software which does similar work (not for profit of course) in a lab at a university in Dallas. Any tips? :)