Self-organizing neural integration of pose-motion features for human action recognition
The visual recognition of complex, articulated human movements is fundamental for a wide range of artificial systems oriented toward human-robot communication, action classification, and action-driven perception. These challenging tasks may generally involve the processing of a huge amount of visual...
| Main Authors: | , , |
|---|---|
| Format: | Online |
| Language: | English |
| Published: |
Frontiers Media S.A.
2015
|
| Online Access: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4460528/ |
| id |
pubmed-4460528 |
|---|---|
| recordtype |
oai_dc |
| spelling |
pubmed-44605282015-06-23 Self-organizing neural integration of pose-motion features for human action recognition Parisi, German I. Weber, Cornelius Wermter, Stefan Neuroscience The visual recognition of complex, articulated human movements is fundamental for a wide range of artificial systems oriented toward human-robot communication, action classification, and action-driven perception. These challenging tasks may generally involve the processing of a huge amount of visual information and learning-based mechanisms for generalizing a set of training actions and classifying new samples. To operate in natural environments, a crucial property is the efficient and robust recognition of actions, also under noisy conditions caused by, for instance, systematic sensor errors and temporarily occluded persons. Studies of the mammalian visual system and its outperforming ability to process biological motion information suggest separate neural pathways for the distinct processing of pose and motion features at multiple levels and the subsequent integration of these visual cues for action perception. We present a neurobiologically-motivated approach to achieve noise-tolerant action recognition in real time. Our model consists of self-organizing Growing When Required (GWR) networks that obtain progressively generalized representations of sensory inputs and learn inherent spatio-temporal dependencies. During the training, the GWR networks dynamically change their topological structure to better match the input space. We first extract pose and motion features from video sequences and then cluster actions in terms of prototypical pose-motion trajectories. Multi-cue trajectories from matching action frames are subsequently combined to provide action dynamics in the joint feature space. Reported experiments show that our approach outperforms previous results on a dataset of full-body actions captured with a depth sensor, and ranks among the best results for a public benchmark of domestic daily actions. Frontiers Media S.A. 2015-06-09 /pmc/articles/PMC4460528/ /pubmed/26106323 http://dx.doi.org/10.3389/fnbot.2015.00003 Text en Copyright © 2015 Parisi, Weber and Wermter. http://creativecommons.org/licenses/by/4.0/ This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms. |
| repository_type |
Open Access Journal |
| institution_category |
Foreign Institution |
| institution |
US National Center for Biotechnology Information |
| building |
NCBI PubMed |
| collection |
Online Access |
| language |
English |
| format |
Online |
| author |
Parisi, German I. Weber, Cornelius Wermter, Stefan |
| spellingShingle |
Parisi, German I. Weber, Cornelius Wermter, Stefan Self-organizing neural integration of pose-motion features for human action recognition |
| author_facet |
Parisi, German I. Weber, Cornelius Wermter, Stefan |
| author_sort |
Parisi, German I. |
| title |
Self-organizing neural integration of pose-motion features for human action recognition |
| title_short |
Self-organizing neural integration of pose-motion features for human action recognition |
| title_full |
Self-organizing neural integration of pose-motion features for human action recognition |
| title_fullStr |
Self-organizing neural integration of pose-motion features for human action recognition |
| title_full_unstemmed |
Self-organizing neural integration of pose-motion features for human action recognition |
| title_sort |
self-organizing neural integration of pose-motion features for human action recognition |
| description |
The visual recognition of complex, articulated human movements is fundamental for a wide range of artificial systems oriented toward human-robot communication, action classification, and action-driven perception. These challenging tasks may generally involve the processing of a huge amount of visual information and learning-based mechanisms for generalizing a set of training actions and classifying new samples. To operate in natural environments, a crucial property is the efficient and robust recognition of actions, also under noisy conditions caused by, for instance, systematic sensor errors and temporarily occluded persons. Studies of the mammalian visual system and its outperforming ability to process biological motion information suggest separate neural pathways for the distinct processing of pose and motion features at multiple levels and the subsequent integration of these visual cues for action perception. We present a neurobiologically-motivated approach to achieve noise-tolerant action recognition in real time. Our model consists of self-organizing Growing When Required (GWR) networks that obtain progressively generalized representations of sensory inputs and learn inherent spatio-temporal dependencies. During the training, the GWR networks dynamically change their topological structure to better match the input space. We first extract pose and motion features from video sequences and then cluster actions in terms of prototypical pose-motion trajectories. Multi-cue trajectories from matching action frames are subsequently combined to provide action dynamics in the joint feature space. Reported experiments show that our approach outperforms previous results on a dataset of full-body actions captured with a depth sensor, and ranks among the best results for a public benchmark of domestic daily actions. |
| publisher |
Frontiers Media S.A. |
| publishDate |
2015 |
| url |
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4460528/ |
| _version_ |
1613233338052509696 |