Defense Event

Automatic Emotion Recognition: Quantifying Dynamics and Structure in Human Behavior

Yelin Kim

Thursday, July 07, 2016
10:00am - 12:00pm
3725 BBB

Add to Google Calendar

About the Event

Emotion is a central part of human interaction that has a huge influence on its overall tone and outcome. Today’s human-centered interactive technology can greatly benefit from automatic emotion recognition, as the extracted affective information can be used to measure, transmit, and respond to user needs. However, developing such systems is challenging since emotional expressions are complex and dynamic in terms of the inherent multimodality between audio and visual expressions, as well as the mixed factors of modulation that arise when a person speaks. To overcome this challenge, this dissertation presents data-driven approaches that can quantify the underlying dynamics and structure in audio-visual affective behavior. The first set of studies discovers that it is crucial to model complex non-linear interactions between audio and visual emotion expressions. This finding leads us to examine how speech modulates facial displays of emotion, which constitutes the central topic of this dissertation. We develop a framework that uses speech signals, which are hypothesized to alter the temporal dynamics of individual facial regions, to temporally segment and classify facial displays of emotion. We also demonstrate that a system that leverages the human evaluator agreement can improve the overall performance. Our experimental results show that the proposed systems not only improve the system performance, but also provide descriptions of how the affective behaviors change over time. We conclude this dissertation with the future directions that will innovate three main research topics: machine adaptation for personalized technology, human-human interaction assistant systems, and human-centered multimedia content analysis.

Additional Information

Sponsor(s): ECE

Faculty Sponsor: Emily Mower Provost

Open to: Public