Deep learning-based audio-visual speech recognition for Bosnian digits
This study presents a deep learning-based solution for audio-visual speech recognition of Bosnian digits. The task posed a challenge due to the lack of an appropriate Bosnian language dataset, and this study outlines the approach to building a new dataset. The proposed solution includes two comp...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Penerbit Universiti Kebangsaan Malaysia
2024
|
| Online Access: | http://journalarticle.ukm.my/25132/ http://journalarticle.ukm.my/25132/1/14.pdf |
| Summary: | This study presents a deep learning-based solution for audio-visual speech recognition of Bosnian digits. The task
posed a challenge due to the lack of an appropriate Bosnian language dataset, and this study outlines the approach to
building a new dataset. The proposed solution includes two components: visual speech recognition, which involves lip reading, and audio speech recognition. For visual speech recognition, a combined CNN-RNN architecture was utilised, consisting of two CNN variants namely Google Net and ResNet-50. These architectures were compared based on their performance, with ResNet-50 achieving 72% accuracy and Google Net achieving 63% accuracy. The RNN component used LSTM. For audio speech recognition, FFT is applied to obtain spectrograms from the input speech signal, which are then classified using a CNN architecture. This component achieved an accuracy of 100%. The dataset was split into three parts namely for training, validation, and testing purposes such that 80%, 10% and 10% of data is allocated to each part, respectively. Furthermore, the predictions from the visual and audio models were combined that yielded 100% accuracy based on the developed dataset. The findings from this study
demonstrate that deep learning-based methods show promising results for audio-visual speech recognition of Bosnian
digits, despite the challenge of limited Bosnian language datasets. |
|---|