Towards uncertainty-aware and label-efficient machine learning of human expressive behaviour
The ability to recognise emotional expressions from non-verbal behaviour plays a key role in human-human interaction. Endowing machines with the same ability is critical to enriching human-computer interaction. Despite receiving widespread attention so far, human-level automatic recognition of affec...
| Main Author: | |
|---|---|
| Format: | Thesis (University of Nottingham only) |
| Language: | English |
| Published: |
2022
|
| Subjects: | |
| Online Access: | https://eprints.nottingham.ac.uk/71876/ |
| _version_ | 1848800699149713408 |
|---|---|
| author | Tellamekala, Mani Kumar |
| author_facet | Tellamekala, Mani Kumar |
| author_sort | Tellamekala, Mani Kumar |
| building | Nottingham Research Data Repository |
| collection | Online Access |
| description | The ability to recognise emotional expressions from non-verbal behaviour plays a key role in human-human interaction. Endowing machines with the same ability is critical to enriching human-computer interaction. Despite receiving widespread attention so far, human-level automatic recognition of affective expressions is still an elusive task for machines. Towards improving the current state of machine learning methods applied to affect recognition, this thesis identifies two challenges: label ambiguity and label scarcity.
Firstly, this thesis notes that it is difficult to establish a clear one-to-one mapping between inputs (face images or speech segments) and their target emotion labels, considering that emotion perception is inherently subjective. As a result, the problem of label ambiguity naturally arises in the manual annotations of affect. Ignoring this fundamental problem, most existing affect recognition methods implicitly assume a one-to-one input-target mapping and use deterministic function learning. In contrast, this thesis proposes to learn non-deterministic functions based on uncertainty-aware probabilistic models, as they can naturally accommodate the one-to-many input-target mapping. Besides improving the affect recognition performance, the proposed uncertainty-aware models in this thesis demonstrate three important applications: adaptive multimodal affect fusion, human-in-the-loop learning of affect, and improved performance on downstream behavioural analysis tasks like personality traits estimation.
Secondly, this thesis aims to address the challenge of scarcity of affect labelled datasets, caused by the cumbersome and time-consuming nature of the affect annotation process. To this end, this thesis notes that audio and visual feature encoders used in the existing models are label-inefficient i.e. learning them requires large amounts of labelled training data. As a solution, this thesis proposes to pre-train the feature encoders using unlabelled data to make them more label-efficient i.e. using as few labelled training examples as possible to achieve good emotion recognition performance. A novel self-supervised pre-training method is proposed in this thesis by posing hand-engineered emotion features as task-specific representation learning priors. By leveraging large amounts of unlabelled audiovisual data, the proposed self-supervised pre-training method demonstrates much better label efficiency compared to the commonly employed pre-training methods. |
| first_indexed | 2025-11-14T20:55:42Z |
| format | Thesis (University of Nottingham only) |
| id | nottingham-71876 |
| institution | University of Nottingham Malaysia Campus |
| institution_category | Local University |
| language | English |
| last_indexed | 2025-11-14T20:55:42Z |
| publishDate | 2022 |
| recordtype | eprints |
| repository_type | Digital Repository |
| spelling | nottingham-718762022-12-15T09:56:33Z https://eprints.nottingham.ac.uk/71876/ Towards uncertainty-aware and label-efficient machine learning of human expressive behaviour Tellamekala, Mani Kumar The ability to recognise emotional expressions from non-verbal behaviour plays a key role in human-human interaction. Endowing machines with the same ability is critical to enriching human-computer interaction. Despite receiving widespread attention so far, human-level automatic recognition of affective expressions is still an elusive task for machines. Towards improving the current state of machine learning methods applied to affect recognition, this thesis identifies two challenges: label ambiguity and label scarcity. Firstly, this thesis notes that it is difficult to establish a clear one-to-one mapping between inputs (face images or speech segments) and their target emotion labels, considering that emotion perception is inherently subjective. As a result, the problem of label ambiguity naturally arises in the manual annotations of affect. Ignoring this fundamental problem, most existing affect recognition methods implicitly assume a one-to-one input-target mapping and use deterministic function learning. In contrast, this thesis proposes to learn non-deterministic functions based on uncertainty-aware probabilistic models, as they can naturally accommodate the one-to-many input-target mapping. Besides improving the affect recognition performance, the proposed uncertainty-aware models in this thesis demonstrate three important applications: adaptive multimodal affect fusion, human-in-the-loop learning of affect, and improved performance on downstream behavioural analysis tasks like personality traits estimation. Secondly, this thesis aims to address the challenge of scarcity of affect labelled datasets, caused by the cumbersome and time-consuming nature of the affect annotation process. To this end, this thesis notes that audio and visual feature encoders used in the existing models are label-inefficient i.e. learning them requires large amounts of labelled training data. As a solution, this thesis proposes to pre-train the feature encoders using unlabelled data to make them more label-efficient i.e. using as few labelled training examples as possible to achieve good emotion recognition performance. A novel self-supervised pre-training method is proposed in this thesis by posing hand-engineered emotion features as task-specific representation learning priors. By leveraging large amounts of unlabelled audiovisual data, the proposed self-supervised pre-training method demonstrates much better label efficiency compared to the commonly employed pre-training methods. 2022-12-14 Thesis (University of Nottingham only) NonPeerReviewed application/pdf en cc_by https://eprints.nottingham.ac.uk/71876/1/Mani_PhD_Thesis_with_Final_Corrections.pdf Tellamekala, Mani Kumar (2022) Towards uncertainty-aware and label-efficient machine learning of human expressive behaviour. PhD thesis, University of Nottingham. Emotion Recognition Machine Learning Uncertainty Modelling Self-supervised Learning Probabilistic Machine Learning Models Stochastic Process Regression Uncertainty-Aware Learning Label Ambiguity |
| spellingShingle | Emotion Recognition Machine Learning Uncertainty Modelling Self-supervised Learning Probabilistic Machine Learning Models Stochastic Process Regression Uncertainty-Aware Learning Label Ambiguity Tellamekala, Mani Kumar Towards uncertainty-aware and label-efficient machine learning of human expressive behaviour |
| title | Towards uncertainty-aware and label-efficient machine learning of human expressive behaviour |
| title_full | Towards uncertainty-aware and label-efficient machine learning of human expressive behaviour |
| title_fullStr | Towards uncertainty-aware and label-efficient machine learning of human expressive behaviour |
| title_full_unstemmed | Towards uncertainty-aware and label-efficient machine learning of human expressive behaviour |
| title_short | Towards uncertainty-aware and label-efficient machine learning of human expressive behaviour |
| title_sort | towards uncertainty-aware and label-efficient machine learning of human expressive behaviour |
| topic | Emotion Recognition Machine Learning Uncertainty Modelling Self-supervised Learning Probabilistic Machine Learning Models Stochastic Process Regression Uncertainty-Aware Learning Label Ambiguity |
| url | https://eprints.nottingham.ac.uk/71876/ |