Improved deepfake video detection using convolutional vision transformer
- Author
- Deressa Wodajo Deressa (UGent) , Peter Lambert (UGent) , Glenn Van Wallendael (UGent) , Solomon Atnafu and Hannes Mareen (UGent)
- Organization
- Project
- Abstract
- Deepfakes are hyper-realistic videos in which the faces are replaced, swapped, or forged using deep-learning models. This potent media manipulation techniques hold promise for applications across various domains. Yet, they also present a significant risk when employed for malicious intents like iden-tity fraud, phishing, spreading false information, and executing scams. In this work, we propose a novel and improved Deepfake video detector that uses a Convolutional Vision Transformer (CViT2), which builds on the concepts of our previous work (CViT). The CViT architecture consists of two components: a Convolutional Neural Network that extracts learnable features, and a Vision Transformer that categorizes these learned features using an attention mechanism. We trained and evaluted our model on 5 datasets, namely Deepfake Detection Challenge Dataset (DFDC), FaceForensics++ (FF++) I, Celeb-DF v2, Deep-fakeTIMIT, and TrustedMedia. On the test sets unseen during training, we achieved an accuracy of 95 %, 94.8 %, 98.3 % and 76.7% on the DFDC, FF++ , Celeb-DF v2, and TIMIT datasets, respectively. In conclusion, our proposed Deepfake detector can be used in the battle against misinformation and other forensic use cases.
- Keywords
- Deepfake Video Detection, Vision Transformer, Convolutional Neural Network, Misinformation Detection, Multimedia Forensics
Downloads
-
DS792 acc.pdf
- full text (Accepted manuscript)
- |
- open access
- |
- |
- 353.93 KB
-
(...).pdf
- full text (Published version)
- |
- UGent only
- |
- |
- 406.00 KB
Citation
Please use this url to cite or link to this publication: http://hdl.handle.net/1854/LU-01J2X7QBWYDBDCG7RA4FYC35Z1
- MLA
- Deressa, Deressa Wodajo, et al. “Improved Deepfake Video Detection Using Convolutional Vision Transformer.” 2024 IEEE GAMING, ENTERTAINMENT, AND MEDIA CONFERENCE, GEM 2024, IEEE, 2024, pp. 492–97, doi:10.1109/gem61861.2024.10585593.
- APA
- Deressa, D. W., Lambert, P., Van Wallendael, G., Atnafu, S., & Mareen, H. (2024). Improved deepfake video detection using convolutional vision transformer. 2024 IEEE GAMING, ENTERTAINMENT, AND MEDIA CONFERENCE, GEM 2024, 492–497. https://doi.org/10.1109/gem61861.2024.10585593
- Chicago author-date
- Deressa, Deressa Wodajo, Peter Lambert, Glenn Van Wallendael, Solomon Atnafu, and Hannes Mareen. 2024. “Improved Deepfake Video Detection Using Convolutional Vision Transformer.” In 2024 IEEE GAMING, ENTERTAINMENT, AND MEDIA CONFERENCE, GEM 2024, 492–97. IEEE. https://doi.org/10.1109/gem61861.2024.10585593.
- Chicago author-date (all authors)
- Deressa, Deressa Wodajo, Peter Lambert, Glenn Van Wallendael, Solomon Atnafu, and Hannes Mareen. 2024. “Improved Deepfake Video Detection Using Convolutional Vision Transformer.” In 2024 IEEE GAMING, ENTERTAINMENT, AND MEDIA CONFERENCE, GEM 2024, 492–497. IEEE. doi:10.1109/gem61861.2024.10585593.
- Vancouver
- 1.Deressa DW, Lambert P, Van Wallendael G, Atnafu S, Mareen H. Improved deepfake video detection using convolutional vision transformer. In: 2024 IEEE GAMING, ENTERTAINMENT, AND MEDIA CONFERENCE, GEM 2024. IEEE; 2024. p. 492–7.
- IEEE
- [1]D. W. Deressa, P. Lambert, G. Van Wallendael, S. Atnafu, and H. Mareen, “Improved deepfake video detection using convolutional vision transformer,” in 2024 IEEE GAMING, ENTERTAINMENT, AND MEDIA CONFERENCE, GEM 2024, Turin, Italy, 2024, pp. 492–497.
@inproceedings{01J2X7QBWYDBDCG7RA4FYC35Z1,
abstract = {{Deepfakes are hyper-realistic videos in which the faces are replaced, swapped, or forged using deep-learning models. This potent media manipulation techniques hold promise for applications across various domains. Yet, they also present a significant risk when employed for malicious intents like iden-tity fraud, phishing, spreading false information, and executing scams. In this work, we propose a novel and improved Deepfake video detector that uses a Convolutional Vision Transformer (CViT2), which builds on the concepts of our previous work (CViT). The CViT architecture consists of two components: a Convolutional Neural Network that extracts learnable features, and a Vision Transformer that categorizes these learned features using an attention mechanism. We trained and evaluted our model on 5 datasets, namely Deepfake Detection Challenge Dataset (DFDC), FaceForensics++ (FF++) I, Celeb-DF v2, Deep-fakeTIMIT, and TrustedMedia. On the test sets unseen during training, we achieved an accuracy of 95 %, 94.8 %, 98.3 % and 76.7% on the DFDC, FF++ , Celeb-DF v2, and TIMIT datasets, respectively. In conclusion, our proposed Deepfake detector can be used in the battle against misinformation and other forensic use cases.}},
author = {{Deressa, Deressa Wodajo and Lambert, Peter and Van Wallendael, Glenn and Atnafu, Solomon and Mareen, Hannes}},
booktitle = {{2024 IEEE GAMING, ENTERTAINMENT, AND MEDIA CONFERENCE, GEM 2024}},
isbn = {{9798350374537}},
issn = {{2831-5510}},
keywords = {{Deepfake Video Detection,Vision Transformer,Convolutional Neural Network,Misinformation Detection,Multimedia Forensics}},
language = {{eng}},
location = {{Turin, Italy}},
pages = {{492--497}},
publisher = {{IEEE}},
title = {{Improved deepfake video detection using convolutional vision transformer}},
url = {{http://doi.org/10.1109/gem61861.2024.10585593}},
year = {{2024}},
}
- Altmetric
- View in Altmetric
- Web of Science
- Times cited: