Using cascade CNN-LSTM-FCNs to identify AI-altered video based on eye state sequence
Deep learning is notably successful in data analysis, computer vision, and human control. Nevertheless, this approach has inevitably allowed the development of DeepFake video sequences and images that could be altered so that the changes are not easily or explicitly detectable. Such alterations have...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science
2022
|
| Subjects: | |
| Online Access: | http://umpir.ump.edu.my/id/eprint/39129/ http://umpir.ump.edu.my/id/eprint/39129/1/journal.pone.0278989%20%282%29.pdf |
| _version_ | 1848825691413413888 |
|---|---|
| author | Muhammad Salihin, Saealal Mohd Zamri, Ibrahim Mulvaney, D. J. Mohd Ibrahim, Shapiai Norasyikin, Fadilah |
| author_facet | Muhammad Salihin, Saealal Mohd Zamri, Ibrahim Mulvaney, D. J. Mohd Ibrahim, Shapiai Norasyikin, Fadilah |
| author_sort | Muhammad Salihin, Saealal |
| building | UMP Institutional Repository |
| collection | Online Access |
| description | Deep learning is notably successful in data analysis, computer vision, and human control. Nevertheless, this approach has inevitably allowed the development of DeepFake video sequences and images that could be altered so that the changes are not easily or explicitly detectable. Such alterations have been recently used to spread false news or disinformation. This study aims to identify Deepfaked videos and images and alert viewers to the possible falsity of the information. The current work presented a novel means of revealing fake face videos by cascading the convolution network with recurrent neural networks and fully connected network (FCN) models. The system detection approach utilizes the eye-blinking state in temporal video frames. Notwithstanding, it is deemed challenging to precisely depict (i) artificiality in fake videos and (ii) spatial information within the individual frame through this physiological signal. Spatial features were extracted using the VGG16 network and trained with the ImageNet dataset. The temporal features were then extracted in every 20 sequences through the LSTM network. On another note, the pre-processed eye-blinking state served as a probability to generate a novel BPD dataset. This newly-acquired dataset was fed to three models for training purposes with each entailing four, three, and six hidden layers, respectively. Every model constitutes a unique architecture and specific dropout value. Resultantly, the model optimally and accurately identified tampered videos within the dataset. The study model was assessed using the current BPD dataset based on one of the most complex datasets (FaceForensic++) with 90.8% accuracy. Such precision was successfully maintained in datasets that were not used in the training process. The training process was also accelerated by lowering the computation prerequisites. |
| first_indexed | 2025-11-15T03:32:57Z |
| format | Article |
| id | ump-39129 |
| institution | Universiti Malaysia Pahang |
| institution_category | Local University |
| language | English |
| last_indexed | 2025-11-15T03:32:57Z |
| publishDate | 2022 |
| publisher | Public Library of Science |
| recordtype | eprints |
| repository_type | Digital Repository |
| spelling | ump-391292023-11-01T07:46:33Z http://umpir.ump.edu.my/id/eprint/39129/ Using cascade CNN-LSTM-FCNs to identify AI-altered video based on eye state sequence Muhammad Salihin, Saealal Mohd Zamri, Ibrahim Mulvaney, D. J. Mohd Ibrahim, Shapiai Norasyikin, Fadilah QA75 Electronic computers. Computer science Deep learning is notably successful in data analysis, computer vision, and human control. Nevertheless, this approach has inevitably allowed the development of DeepFake video sequences and images that could be altered so that the changes are not easily or explicitly detectable. Such alterations have been recently used to spread false news or disinformation. This study aims to identify Deepfaked videos and images and alert viewers to the possible falsity of the information. The current work presented a novel means of revealing fake face videos by cascading the convolution network with recurrent neural networks and fully connected network (FCN) models. The system detection approach utilizes the eye-blinking state in temporal video frames. Notwithstanding, it is deemed challenging to precisely depict (i) artificiality in fake videos and (ii) spatial information within the individual frame through this physiological signal. Spatial features were extracted using the VGG16 network and trained with the ImageNet dataset. The temporal features were then extracted in every 20 sequences through the LSTM network. On another note, the pre-processed eye-blinking state served as a probability to generate a novel BPD dataset. This newly-acquired dataset was fed to three models for training purposes with each entailing four, three, and six hidden layers, respectively. Every model constitutes a unique architecture and specific dropout value. Resultantly, the model optimally and accurately identified tampered videos within the dataset. The study model was assessed using the current BPD dataset based on one of the most complex datasets (FaceForensic++) with 90.8% accuracy. Such precision was successfully maintained in datasets that were not used in the training process. The training process was also accelerated by lowering the computation prerequisites. Public Library of Science 2022 Article PeerReviewed pdf en cc_by_4 http://umpir.ump.edu.my/id/eprint/39129/1/journal.pone.0278989%20%282%29.pdf Muhammad Salihin, Saealal and Mohd Zamri, Ibrahim and Mulvaney, D. J. and Mohd Ibrahim, Shapiai and Norasyikin, Fadilah (2022) Using cascade CNN-LSTM-FCNs to identify AI-altered video based on eye state sequence. PLoS ONE, 17 (12). ISSN 1932-6203. (Published) https://doi.org/10.1371/journal.pone.0278989 10.1371/journal.pone.0278989 |
| spellingShingle | QA75 Electronic computers. Computer science Muhammad Salihin, Saealal Mohd Zamri, Ibrahim Mulvaney, D. J. Mohd Ibrahim, Shapiai Norasyikin, Fadilah Using cascade CNN-LSTM-FCNs to identify AI-altered video based on eye state sequence |
| title | Using cascade CNN-LSTM-FCNs to identify AI-altered video based on eye state sequence |
| title_full | Using cascade CNN-LSTM-FCNs to identify AI-altered video based on eye state sequence |
| title_fullStr | Using cascade CNN-LSTM-FCNs to identify AI-altered video based on eye state sequence |
| title_full_unstemmed | Using cascade CNN-LSTM-FCNs to identify AI-altered video based on eye state sequence |
| title_short | Using cascade CNN-LSTM-FCNs to identify AI-altered video based on eye state sequence |
| title_sort | using cascade cnn-lstm-fcns to identify ai-altered video based on eye state sequence |
| topic | QA75 Electronic computers. Computer science |
| url | http://umpir.ump.edu.my/id/eprint/39129/ http://umpir.ump.edu.my/id/eprint/39129/ http://umpir.ump.edu.my/id/eprint/39129/ http://umpir.ump.edu.my/id/eprint/39129/1/journal.pone.0278989%20%282%29.pdf |