2009

2008

2007

2006

2005

Before 2005

"Style Learning and Transferring for Facial Animation Editing,"GF
X. Ma*, B. H. Le*, and Z. Deng,
Proc. of ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA) 2009, New Orleans, LA, August 1-2, 2009, (to appear).
We present a novel facial editing style learning framework that is able to learn a constraint-based Gaussian Process model from a small number of facial-editing pairs, and then it can be effectively applied to automate the editing of the remaining facial animation frames or transfer editing styles between different animation sequences.

"Intermittency of Slow Arm Movements Increases in Distal Direction,"H
O. Celik, Q. Gu*, Z. Deng, and M. O'Malley,
Proc. of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2009, St Louis, MO, Oct 11-15, 2009, (to appear).
When analyzed in the tangential speed domain, human movements exhibit a multi-peaked speed profile which is commonly interpreted as evidence for submovements. At slow speeds, the number of the peaks increases and the peaks also become more distinct, corresponding to non-smoothness or intermittency in the movement. In this study, we evaluate two potential sources proposed in the literature for the origins of movement intermittency and conclude that intermittency is more likely due to noise in the neuromuscular system as opposed to a central movement planner that generates intermittent plans.

"An Interactive Geometrical Technique for Upper and Lower Teeth Segmentation,"GV
B. H. Le*, Z. Deng, J. Xia, Y.-B. Chang, and X. Zhou,
Proc. of International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) 2009, Lodon, UK, Sep 20-24, 2009, (to appear). [Demo Video]
We present a novel interactive method to segment the upper and lower teeth. The process is performed on 3D triangular mesh of the skull and consists of four main steps: reconstruction of 3D model from teeth CT images, curvature estimation, interactive segmentation path planning using the shortest path finding algorithm, and performing actual geometric cut on 3D models using a graph cut algorithm. The accuracy and efficiency of our method were experimentally validated via comparisons with ground truth (manual segmentation) as well as the state of art interactive mesh segmentation algorithms.

"Image-based Face Illumination Transferring using Logarithmic Total Variation Models,"G
Q. Li*, W. Yin, and Z. Deng,
The Visual Computer (TVCJ), May 2009, (in press).
Also published as a UH CS Technical Report UH-CS-09-06.
We propose a novel image-based technique that transfers illumination from a source face image to a target face image based on the Logarithmic Total Variation (LTV) model. Our method does not require any prior information regarding the lighting conditions or the 3D geometries of the underlying faces.

"A Novel Surface-based Geometric Approach for 3D Dentritic Spline Detection From Multi-Photon Excitation Microscopy Images,"V
Q. Li*, X. Zhou, Z. Deng, M. Baron, M. A. Teylan, Y. Kim, S. Wong,
Proc. of IEEE International Symposium on Biomedical Imaging (ISBI) 2009, Boston, MA, June 2009, (to appear).
Determining the relationship between the dendritic spine morphology and its functional properties is a fundamental while challenging problem in neurobiology research. In particular, how to accurately and automatically analyze meaningful structural information from a large microscopy image dataset is far away from being resolved. In this paper, we propose a novel method for the automated neuron reconstruction and spine detection from fluorescence microscopy images.

"Improving Gaming Experience Using Real-Time, Adaptive Game Difficulty Adjustments by Utilizing Physiological Feedback,"H
C. Yun*, (co-advisors: Z. Deng and O. Johnson)
Ph.D. Dissertation, Department of Computer Science, University of Houston, April 2009.
This dissertation introduces a novel methodology that improves the individual users' gaming experiences by automatically adjusting the game difficulty continuously, based on the users' physiological feedbacks. The physiological data of the player - blood flow in the supraorbital region - is continuously monitored using StressCam, a contact-free, real-time, thermal image-based monitoring and analysis system. This stress level is utilized in adjusting the game difficulty to satisfy the player throughout the game. Extensive user study results show successful real-time automatic game difficulty adjustments based on corresponding physiological feedbacks from the gamers. Furthermore, the results also demonstrate an increase in gamer performances as well as improvements in subjective ratings of perceptions of the game difficulty and the game entertainment values. We conclude that our user-centric approach successfully adjusted game difficulty automatically to accommodate the individual gamers and, consequently, improved gaming experience of the gamers who had individually unique gaming expertise or preference.

"Can Local Avatars Satisfy Global Audience? A Case Study of High-Fidelity 3D Facial Avatar Animation in Subject Identification and Emotion Perception by US and International Groups,"H
C. Yun*, Z. Deng, and M. Hiscock,
ACM Computer In Entertainment (CIE), 7(2), April/June 2009, pp. 1-26. [bibtex]
Also published as UH CS Technical Report UH-CS-09-05.
This paper investigates the effectiveness of a local high-fidelity 3D facial avatar before the global audience by observing how US and International student groups differ in viewing non-verbal high fidelity 3D facial avatar animation embedded with motion data from 3 US individuals in subject identification and emotion perception. The results of our study provide interesting insights to avatar-based interaction where national/cultural background of a person impacts the perception of the identity while it has little effect on the emotion perception.

"O' Game, Can You Feel My Frustration?: Improving User's Gaming Experience via StressCam," H
C. Yun*, D. Shastri, I. Pavlidis, and Z. Deng
Proc. of ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) 2009, Boston, MA, April 5-9, 2009, pp. 2195-2204. (full paper). [bibtex]
This paper presents a novel methodology to improve users' experience in computer games by automatically adjusting the level of the game difficulty. The difficulty level is computed from measurements of the facial physiology of the players at a distance. players' performance during the game-playing session alters blood flow in the supraorbital region, which is an indirect measurement of increased mental activities. Our results show that the automatic difficulty adjustable system successfully maintains game players' interests and substantially outperforms traditiona fixed-difficulty mode games.

"Perceptually Consistent Example-based Human Motion Retrieval," GC
Z. Deng, Q. Gu* , and Q. Li*
Proc. of ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (SI3D) 2009, Boston, MA, Feb 27-March 1, 2009, pp. 191-198. [bibtex], [Demo Video]
This paper presents a perceptually consistent, example-based human motion retrieval approach that is capable of efficiently searching for and ranking similar motion sequences given a query motion input. Our method employs a motion pattern discovery and matching scheme that breaks human motions into a part-based, hierarchical motion representation. Building upon this representation, a fast string match algorithm is used for efficient runtime motion query processing. Finally, we conducted comparative user studies to evaluate the accuracy and perceptualconsistency of our approach by comparing it with the state of the art example-based human motion search algorithms.

"Natural Eye Motion Synthesis by Modeling Gaze-Head Coupling," GF
X. Ma* and Z. Deng,
Proc. of IEEE Virtual Reality (VR) 2009 Conference, Lafayette, Louisiana, March 14-18, 2009, pp. 143-150. [bibtex], [Demo Video (xvid codec)]
This paper presents an effective technique to synthesize natural eye gazes given a head motion sequence as input, by statistically modeling (i.e., dynamic coupled component analysis model) the innate coupling between gazes and head movements. Through comparative user studies and evaluations, we found that comparing with the state of the art algorithms in eye motion synthesis, our approach is more effective to generate natural gazes correlated with given head motions. We also showed the effectiveness of our approach for gaze simulation in two-party conversations.

"Crafting Personalized Facial Avatars Using Editable Portrait and Photograph Example,"GF
T. Sucontphunt*, Z. Deng, and U. Neumann,
Proc. of IEEE Virtual Reality (VR) 2009 Conference, Lafayette, Louisiana, March 14-18, 2009, pp.259-260. [bibtex],
This poster paper presents an interactive system to intuitively craft personalized 3D facial avatars by using 3D portrait editing and image example-based painting techniques. Starting from a default 3D face portrait, users can conveniently perform intuitive pulling operations on its 3D surface to sculpt the 3D face shape towards any individual.

"Compression of Human Motion Capture Data using Motion Pattern Indexing," GC
Q. Gu*, J. Peng, and Z. Deng,
Computer Graphics Forum (CGF), 28(1), 2009, pp. 1-12. [bibtex],[Demo Video]
Recommended by the Eurographics 2008 Paper Committee. Also published as UH CS Technical Report UH-CS-08-13.
This paper proposes a novel scheme to compress human motion capture data based on hierarchical structure construction and motion pattern indexing. Its central scheme is to greatly reduces the redundancy in a motion sequence by utilizing the fact that the motion of a meaningful human part (e.g. arm, leg, etc.) usually exhibits similar patterns over the time. As a result, higher compression ratio has been achieved when compared with the prior art, especially for long sequences of motion capture data with repetitive motion styles. Another distinction of this work is that it provides means for flexible and intuitive global and local distortion controls.

2008

"Orthogonal Blendshape based Editing System for Facial Motion Capture Data,"GF
Q. Li* and Z. Deng,
IEEE Computer Graphics and Applications (CG&A), 28(6), Nov/December, 2008, pp. 76-82. [bibtex], [Demo Video]
Also published as UH CS Technical Report UH-CS-07-12.
Aiming to bridge the popularized facial motion capture technique and blendshape approach, this paper presents a novel data-driven 3D facial motion capture data editing system that essentially converts 3D facial motion capture editing problem is transformed to a blendshape animation editing problem through the automated construction of an orthogonal blendshape face model. It is the official published version of UH-CS-07-12.

"Facial and Character Animation Research at UH CGIM," O
Z. Deng, Q. Gu*, Q.Li*, X. Ma*, and T. Sucontphunt*,
ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA) 2008 Poster, Dublin, Ireland, July 7-9, 2008. pp. 33-34, (lab poster). [Demo Video (Divx)]
This poster summerizes the recent research efforts on computer animation, facial animation, and character animation at the UH Computer Graphics and Interactive Media Lab during the past two years.

"Perceptually Guided Expressive Facial Animation," GF
Z. Deng and X. Ma*,
Proc. of ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA) 2008, Dublin, Ireland, July 7-9, 2008, pp. 67-76. [bibtex], [Demo Video]
This paper presents a novel perceptually guided computational framework for expressive facial animation by bridging objective facial motion patterns with subjective perceptual outcomes. Its core piece is the construction of a facial perceptual metric (FacePEM) based on a hybrid of region-based facial motion analysis and statistical learning techniques. The constructed FacePEM model can automatically measure the emotional expressiveness of a facial motion sequence. We showed how the constructed FacePEM model can be effectively incorporated into various facial animation algorithms including data-driven expressive speech animation generation and expressive facial motion editing.

"Expressive Speech Animation Synthesis with Phoneme-Level Controls,"GF
Z. Deng and U. Neumann,
Computer Graphics Forum (CGF), 27(8), 2008, pp. 2096-2113.[bibtex]
Also published as UH CS Technical Report, UH-CS-08-05.
This paper presents a data-driven expressive speech animation synthesis system with phoneme-level controls. Given new phoneme-aligned expressive speech and its emotion modifiers as inputs, a constrained dynamic programming algorithm is used to search for best-matched captured motion clips from the processed facial motion database by minimizing a cost function. Users optionally specify "hard constraints" (motion-node constraints for expressing phoneme utterances) and "soft constraints" (emotion modifiers) to guide this search process. This is an expanded (journal) version of the preliminary SCA06 paper.

"Interactive 3D Facial Expression Posing through 2D Portrait Manipulation," GF
T. Suconphunt*, Z. Mo, U. Neumann, and Z. Deng,
Proc. of Graphics Interface (GI) 2008, Windsor, Ontario, Canada, May 28-30, 2008, pp. 177-184. Selected as one of the cover images of the proceeding. [bibtex],[Demo Video]
This paper presents an interactive 3D facial expression posing system through 2D portrait manipulation, where a manipulated 2D portrait serves a metaphor for automatically inferring its corresponding 3D facial expression with fine details. Users either rapidly assemble a face portrait through a pre-designed portrait component library or intuitively modify an initial portrait. We showed that this system is effective for rapid 3D facial expression sculpting through a comparative user study.

"Facial Motion Capture Editing by Automated Orthogonal Blendshape Construction and Weight Propagation,"GF
Q. Li* and Z. Deng,
Technical Report UH-CS-07-12, Computer Science Department, University of Houston. [Demo Video].
Aiming to bridge the popularized facial motion capture technique and blendshape approach, this paper presents a novel data-driven 3D facial motion capture data editing system that essentially converts 3D facial motion capture editing problem is transformed to a blendshape animation editing problem through the automated construction of an orthogonal blendshape face model.

"A Novel Visual System for Expressive Facial Motion Data Exploration,"V
T. Sucontphunt*, X. Yuan, Q. Li* and Z. Deng,
Proc. of 2008 IEEE Pacific Visualization Symposium (PacificVis),Kyoto, Japan, March 5-7, 2008. pp. 103-110. [bibtex],[Demo Video]
This paper presents a novel visual system for expressive facial motion data exploration. Its basic scheme is to transform original high dimensional facial movements to the 2D screen space for intuitive navigation and exploration through region-based dimensionality reduction. The applications of this visualization system for novel facial motion synthesis and emotion recognition are demonstrated.

2007

"Computer Facial Animation: A Survey,"GF
Z. Deng and J. Y. Noh,
book chapter, in "Data-Driven 3D Facial Animation" (Springer, 2007).
This book chapter surveys current facial modeling and animation techniques in the following categories: blend shape or shape interpolation, parameterizations, Facial Action Coding System based approaches, deformation based approaches, physics based muscle modeling, 3D face modeling, performance driven facial animation, MPEG-4 facial animation, visual speech animation, facial animation editing, facial animation transferring, and facial gesture generation.

"Learning expressive human-like head motion sequences from speech,"GF
C. Busso, Z. Deng, U. Neumann and S. Narayanan,
book chapter, in "Data-Driven 3D Facial Animation" (Springer, 2007).
This book chapter is a revised/upgraded version of our CAVW05 head motion work.

[Book] "Data-Driven 3D Facial Animation,"GF
Editors: Z. Deng and U. Neumann,
Springer-Verlag Press, October 2007. ISBN: 978-1-84628-906-4. [bibtex]
This edited book describes the emerging data-driven facial animation techniques developed over the last ten years or so. Its scope is comprehensive, since it covers not only traditional lip-sync (speech animation), but also expressive facial motion, facial gestures, facial modeling, editing and sketching, and facial animation transferring.

"Online Motion Capture Marker Labeling for Multiple Interacting Articulated Targets,"GC
Q. Yu, Q. Li*, and Z. Deng,
Computer Graphics Forum (Proc. of Eurographics 2007), Prague, Czech Republic, September 2007, pp. 477-483.
Also published as UH CS Technical Report UH-CS-07-11. [video], [bibtex]
This paper proposes an online motion capture marker labeling approach for multiple interacting articulated targets. Its main idea is to fit rigid bodies and exploit trained structure and motion models. We showed the effectiveness, efficiency, and robustness of our approach.

"EduNET: A Prototype Educational System Based on Handwritten and Verbal Interaction for Individuals with Multiple Sclerosis," H
C. Yun*, Z. Deng, O. Johnson, and O. Criner,
Proc. of 2007 Rehabilitation Engineering & Assistive Technology Society of North America Annual Conference (RESNA), Phoenix, Arizona, June 2007. [bibtex]
This paper presents a prototype educational system as an intelligent tutorial system for individuals having multiple sclerosis in various ranges and stages of symptoms, by integrating handwriting, speech recognition and speech synthesis technologies.

"Rigid Head Motion in Expressive Facial Animation: Analysis and Synthesis," F
C. Busso, Z. Deng, M. Grimm, U. Neumann, and S. Narayanan,
IEEE Transaction on Audio, Speech and Language Processing (TASLP) ,March 2007, 15(3), pp. 1075-1086. [bibtex]
This paper investigates the synthesis and analysis of emotional head motions. An approach to synthesize emotional head motion sequences driven by prosodic features is presented, expanding upon our CAVW05 framework on head motion synthesis. On the analysis side, head motion sequences in expressive facial animations are analyzed in terms of their naturalness and emotional salience in perception. Our results show that head motion provides discriminating information about emotional categories, and head motion modifies the emotional perception of the facial animation especially in the valence and activation domain.

2006

"A Methodology for Mapping Mental Spaces of Game Gentres,"O
J.P. Lewis, P. Fox, and Z. Deng,
ACM SIGGRAPH Symposium on Video Games 2006 (poster), Boston, MA, July 2006.
This poster proposes a methodology for applying manifold learning to subjective similarity ratings over the space of games titles. The resulting concept map shows clustering of related games into spontaneously arising genres.

"eFASE: Expressive Facial Animation Synthesis and Editing with Phoneme-Isomap Controls," GF
Z. Deng and U. Neumann
Proc. of ACM SIGGRAPH/EG Symposium on Computer Animation (SCA) 2006 , Vienna, Austria, Sep 2-4, 2006. pp. 251-259. [video] [bibtex]
This paper presents an expressive data-driven facial animation synthesis system where a constrained dynamic programming algorithm is used to search for best-matched captured motion nodes by minimizing a cost function. Users optionally specify "hard constraints" (motion-node constraints for expressing phoneme utterances) and "soft constraints" (emotion modifiers) to guide the search process.

"Perceiving Visual Emotions with Speech" FH
Z. Deng, J. Bailenson, J.P. Lewis, and U. Neumann,
Proc. of the 6th Int'l Conf. on Intelligent Virtual Agents (IVA) 2006 , Marina Del Rey, CA, Springer-Verlag LNAI (Lecture Notes in Artificial Intelligence), Volume 4133, August 2006, pp. 107-120. [bibtex]
This paper presents the first multifaceted study for evaluating the relation between objective and subjective visual perception of emotion as displayed on a speaking human face, using both full video and sparse point-rendered representations of the face.

"Expressive Facial Animation Synthesis by Learning Speech Co-Articulation and Expression Spaces" GF
Z. Deng, U. Neumann, J.P. Lewis, T.Y. Kim, M. Bulut, and S. Narayanan,
IEEE Transaction on Visualization and Computer Graphics (TVCG), 12(6), Nov/Dec 2006, pp. 1523-1534. [video I],[video II], [bibtex]
This paper presents an expressive facial animation synthesis system enabled by automated learning from facial motion capture data. It includes two parts: the construction of a Phoneme-Independent Expression Eigenspace (PIEES) that encloses the dynamic expression signals and the learning of accurate speech co-articulation model. It is an expanded (journal) version of CGI05 and CASA04 papers.

"Animating Blendshape Faces by Cross-Mapping Motion Capture Data" GF
Z. Deng, P.Y. Chiang, P. Fox, and U. Neumann,
Proc. of ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (SI3D) 2006, Redwood City, CA, March 2006, pp. 43-48. [bibtex]
This paper presents a semi-automatic technique to directly animate popularized 3D blendshape face models by mapping facial motion capture data spaces to 3D blendshape face spaces. Its central idea is to create a RBF based regression model for this mapping. Our results showed that this technique is efficient to animate blendshape face models while offering its generality and flexiblity.

2005

"Reducing Blendshape Interference by Selected Motion Attenuation"GF
J.P. Lewis, J. Mooser, Z. Deng, and U. Neumann,
Proc. of ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (SI3D), Washington DC, April 2005, pp.25-29. [video I], [bibtex]
A major problem in creating blendshape animation is that of blendshape interference: the adjustment of a single blendshape slider may degrade the effects obtained with previous slider movements, because the blendshapes have overlapping, non-orthogonal effects. This paper presents an efficient technique for automatically reducing blendshape interferences by temporarily designating (tagging) a set of points as representatives of the previous (desired) adjustments. The resulting algorithm is easy to implement and demonstrably reduces cases of blendshape interference found in existing models.

"Natural Head Motion Synthesis driven by Acoustic Prosodic Features"GF
C. Busso, Z. Deng, U. Neumann, and S. Narayanan,
Computer Animation and Virtual Worlds (CAVW), 16(3-4), July 2005, pp. 283-290, special issue for selected papers of IEEE Computer Animaton and Social Agents (CASA) 2005. [video I], [bibtex]
This paper presents a novel data-driven approach to synthesize appropriate head motions by sampling from Hidden Markov Models (HMMs) that were trained using a specifically recorded facial motion dataset. Comparison experiments and novel synthesis results show that synthesized head motions follow the temporal dynamic behavior of real human subjects.

"Synthesizing Speech Animation by Learning Compact Speech Co-Articulation Models"GF
Z. Deng, J.P. Lewis, and U. Neumann,
Proc. of IEEE Computer Graphics International (CGI) 2005 , NY, June 2005, pp.19-25. [bibtex]
This paper presents a technique to learn visual speech co-articulation models from 3D facial motion capture data. Its essential idea to fit cubic curves to parameterize motion transition during the visual display of any specific diphone or triphone.

Before 2005

"Audio-based Head Motion Synthesis for Avatar-based Telepresence Systems,"GF
Z. Deng, C. Busso, S. Narayanan, and U. Neumann,
Proc. of ACM SIGMM Workshop on Effective Telepresence (ETP) 2004, NY, Oct 2004, pp.24-30.[bibtex]
This paper presents a data-driven head motion synthesis technique driven by audio. Its essential idea is to use a KNN-based dynamic programming algorithm to select optimal head poses from a pre-recorded dataset.

"Analysis of Emotion Recognition using Facial Expressions, Speech and Multimodal Information"H
C. Busso, Z. Deng, S. Yildirim, M. Bulut, C.M. Lee, A. Kazemzaeh, S. Lee, U. Neumann, and S. Narayanan,
Proc. of ACM Int'l Conf. on Multimodal Interfaces (ICMI), State College, PA, pp.205-211. [bibtex]
This paper analyzes the strengths and the limitations of systems based only on facial expressions or acoustic information. The results reveal that the system based on facial expression gave better performance than the system based on just acoustic information for the emotions considered. Results also show the complementarily of the two modalities and that when these two modalities are fused, the performance and the robustness of the emotion recognition system improve measurably.

"Acoustic Study of Emotions Expressed in Speech," H
S. Yildirim, M. Bulut, C.M. Lee, A. Kazemzadeh, C. Busso, Z. Deng, S. Lee, and S. Narayanan,
Proc. of Int'l Conf. on Spoken Language Processing (ICSLP) 2004, Jeju Island, Korean, Oct 2004, pp. 2193-2196. [bibtex]
This paper attempt to obtain detailed acoustic knowledge on how speech is modulated when speaker's emotion changes from neutral to a certain emotional state. Results show that happiness/anger and neutral/sadness share similar acoustic properties in the speaker. Speech associated with anger and happiness are characterized by longer utterance duration, shorter inter-word silence, higher pitch and energy values with wider ranges, showing the characteristics of exaggerated or hyperarticulated speech.

"Emotion Recognition based on Phoneme Classes,"H
C.M. Lee, S. Yildirim, M. Bulut, A. Kazemazadeh, C. Busso, Z. Deng, S. Lee, and S. Narayanan,
Proc. of Int'l Conf. on Spoken Language Processing (ICSLP) 2004, Jeju Island, Korean, Oct 2004, pp. 889-892. [bibtex]
This paper investigates the usefulness of HMM-based, phoneme-level modeling for the recognition of emotional states from acoustic speech signals. The experiment results showed that the best performance can be obtained by using phoneme-class classifiers over generic emotional HMM classifier and classifiers based on global prosodic features.

"Practical Eye Movement Model using Texture Synthesis"GF
Z. Deng, J.P. Lewis, and U. Neumann,
ACM SIGGRAPH 2003 Technical Sketches, San Diego, CA, July 2003. [bibtex]
This one-page sketch presents a highly-efficient and practical technique for generating realistic eye motions by extending texture synthesis algorithms that were originally used in 2D image/texture applications.

Copyright Notice