Emotion Recognition

A Pattern Analysis Approach

Amit Konar, Aruna Chakraborty

• Publisher: Wiley
• Year: 2014
• Print ISBN: 9781118130667
• eBook ISBN: 9781118910603
• Edition: 1
• Topic: Technology & Engineering
• Subtopic: Computer Vision & Pattern Recognition
• Index: Technology & Engineering

A timely book containing foundations and current research directions on emotion recognition by facial expression, voice, gesture and biopotential signals

This book provides a comprehensive examination of the research methodology of different modalities of emotion recognition. Key topics of discussion include facial expression, voice and biopotential signal-based emotion recognition. Special emphasis is given to feature selection, feature reduction, classifier design and multi-modal fusion to improve performance of emotion-classifiers.

Written by several experts, the book includes several tools and techniques, including dynamic Bayesian networks, neural nets, hidden Markov model, rough sets, type-2 fuzzy sets, support vector machines and their applications in emotion recognition by different modalities. The book ends with a discussion on emotion recognition in automotive fields to determine stress and anger of the drivers, responsible for degradation of their performance and driving-ability.

There is an increasing demand of emotion recognition in diverse fields, including psycho-therapy, bio-medicine and security in government, public and private agencies. The importance of emotion recognition has been given priority by industries including Hewlett Packard in the design and development of the next generation human-computer interface (HCI) systems.

Emotion Recognition: A Pattern Analysis Approach would be of great interest to researchers, graduate students and practitioners, as the book

• Offers both foundations and advances on emotion recognition in a single volume
• Provides a thorough and insightful introduction to the subject by utilizing computational tools of diverse domains
• Inspires young researchers to prepare themselves for their own research
• Demonstrates direction of future research through new technologies, such as Microsoft Kinect, EEG systems etc.

1
INTRODUCTION TO EMOTION RECOGNITION

AMIT KONAR AND ANISHA HALDER

Artificial Intelligence Laboratory, Department of Electronics and Telecommunication Engineering, Jadavpur University, Kolkata, India

ARUNA CHAKRABORTY

Department of Computer Science & Engineering, St. Thomas’ College of Engineering & Technology, Kolkata, India

A pattern represents a characteristic set of attributes of an object by which it can be distinguished from other objects. Pattern recognition aims at recognizing an object by its characteristic attributes. This chapter examines emotion recognition in the settings of pattern recognition problems. It begins with an overview of the well-known pattern recognition techniques, and gradually demonstrates the scope of their applications in emotion recognition with special emphasis on feature extraction, feature reduction, and classification. Main emphasis is given to feature selection by single and multiple modalities and classification by neural, fuzzy, and statistical pattern recognition techniques. The chapter also provides an overview of stimulus generation for arousal of emotion. Lastly, the chapter outlines the methods of performance analysis and validation issues in the context of emotion recognition.

1.1 BASICS OF PATTERN RECOGNITION

A pattern is a representative signature of an object by which we can recognize it easily. Pattern recognition refers to mapping of a set of patterns into one of several object classes. Occasionally, a pattern is represented by a vector containing the features of an object. Thus, in general, the pattern recognition process can be described by three fundamental steps, namely, feature extraction, feature selection, and classification. Figure 1.1 provides a general scheme for pattern recognition. The feature extraction process involves using one or more sensors to measure the representative features of an object. The feature selection module selects more fundamental features from a list of features. The classification module classifies the selected features into one of several object classes.

The pattern recognition problem can be broadly divided into two main heads: (i) supervised classification (or discrimination), and (ii) unsupervised clustering. In supervised classification, usually a set of training instances (or data points) comprising a set of measurements about each object along with its class is given. These data points with their class labels are used as exemplars in the classifier design. Given a data point with unknown class, the classifier once trained with the exemplary instances is able to determine the class label of the given data point. The classifier thus automatically maps an unknown data point to one of several classes using the background knowledge about the exemplary instances.

FIGURE 1.1 Basic steps of pattern recognition.

Beginners to the subject often are confronted with the question: how does the classifier automatically determine the class label of an unknown data point, which is not present in the exemplary instances. This is due to the inherent generalization characteristics of the supervised classifier.

In unsupervised classification, the class labels of the data points are not known. The learning system partitions the whole set of data points into (preferably) nonoverlapping subsets based on some measure of similarity of the data points under each subset. Each subset is called a class/cluster. Because of its inherent characteristics of grouping data points into clusters, unsupervised classification is also called clustering.

Both statistical decision theory and machine learning have been employed in the literature to design pattern recognition algorithms [1, 2]. Bayes’ theorem is the building stone of statistical classification algorithms. On the other hand there exists a vast literature on supervised and unsupervised learning [3], algorithms, which capture the inherent structural similarity [4] of the data points for application in pattern recognition problems.

1.2 EMOTION DETECTION AS A PATTERN RECOGNITION PROBLEM

Emotion represents the psychological state of the human mind and thought processes. Apparently, the process of arousal of emotion has a good resemblance with its manifestation as facial, vocal, and bodily gestures. This phenomenon has attracted researchers to determine the emotion of a subject from its manifestation. Although the one-to-one correspondence from manifestation of emotion to a particular emotional state is yet to be proved, researchers presume the existence of such mapping to recognize the emotion of a subject from its manifestation.

Given the manifestation of an emotion, the task of recognizing the emotion, thus, is a pattern recognition problem. For example, facial expression–based emotion recognition requires extraction of a set of facial features from the facial expression of a given subject. Recognition of emotion here refers to classification of facial features into one of several emotion classes. Usually, a supervised classifier pretrained with emotional features as input and emotion class as output is used to determine the class of an unknown emotional manifestation.

Apparently, the emotional state of the human mind is expressed in different modes including facial, voice, gesture, posture, and biopotential signals. When a single mode of manifestation is used to recognize emotion, we call it a unimodal approach. Sometimes all modes are not sufficiently expressed. Naturally, recognition from a less expressed mode invites the scope of misclassification. This problem can be avoided by attempting to recognize an emotion from several modalities. Such a process is often referred to as multimodal emotion recognition.

1.3 FEATURE EXTRACTION

Feature extraction is one of the fundamental steps in emotion recognition. Features are obtained in different ways. On occasion features are preprocessed sensory readings. Preprocessing is required to filter noise from measurements. Sensory readings during the period of emotion arousal sometimes have a wide variance. Statistical estimates of the temporal readings, such as mean, variance, skewness, kurtosis, and the like, are usually taken to reduce the effect of temporal variations on measurements. Further, instead of directly using time/spatial domain measurements, frequency domain transforms are also used to extract frequency domain features. For example, frequency domain information is generally used for EEG (electroencephalogram) and voice signals. Frequency domain parameters are time invariant and less susceptible to noise. This attracted researchers to use frequency domain features instead of time domain.

Frequency domain features have one fundamental limitation in that they are unable to tag time with frequency components. Tagging the time with frequency contents of a signal is important, particularly for a certain class of signals, often labeled as nonstationary signals. EEG, for instance, is a nonstationary signal, the frequency contents...

Table of contents

• Cover
• Title Page
• Copyright
• Dedication
• Preface
• Acknowledgments
• Contributors
• Chapter 1: Introduction to Emotion Recognition
• Chapter 2: Exploiting Dynamic Dependencies Among Action Units for Spontaneous Facial Action Recognition
• Chapter 3: Facial Expressions: A Cross-Cultural Study
• Chapter 4: A Subject-dependent Facial Expression Recognition System
• Chapter 5: Facial Expression Recognition Using Independent Component Features and Hidden Markov Model
• Chapter 6: Feature Selection for Facial Expression based on Rough Set Theory
• Chapter 7: Emotion Recognition from Facial Expressions Using Type-2 Fuzzy Sets
• Chapter 8: Emotion Recognition from Non-frontal Facial Images
• Chapter 9: Maximum a Posteriori based Fusion Method for Speech Emotion Recognition
• Chapter 10: Emotion Recognition in Naturalistic Speech and Language—A Survey
• Chapter 11: EEG-Based Emotion Recognition Using Advanced Signal Processing Techniques
• Chapter 12: Frequency Band Localization on Multiple Physiological Signals for Human Emotion Classification Using DWT
• Chapter 13: Toward Affective Brain–Computer Interface: Fundamentals and Analysis of EEG-based Emotion Classification
• Chapter 14: Bodily Expression for Automatic Affect Recognition
• Chapter 15: Building a Robust System for Multimodal Emotion Recognition
• Chapter 16: Semantic AudioVisual Data Fusion for Automatic Emotion Recognition
• Chapter 17: A Multilevel Fusion Approach for Audiovisual Emotion Recognition
• Chapter 18: From A Discrete Perspective of Emotions to Continuous, Dynamic, and Multimodal Affect Sensing
• Chapter 19: AudioVisual Emotion Recognition using Semi-Coupled Hidden Markov Model with State-Based Alignment Strategy
• Chapter 20: Emotion Recognition in Car Industry
• Index
• End User License Agreement