Combining EfficientNet and Vision Transformers for Video Deepfake Detection
This repository provides the implementation of two deep learning architectures designed for video deepfake detection: Efficient Vision Transformers (Efficient ViT) and Cross Efficient Vision Transformers (Cross Efficient ViT). The work was originally presented at the International Conference on Image Analysis and Processing (ICIAP) in 2021 and later published in the proceedings of ICIAP 2022. The research combines the efficiency of EfficientNet with the transformative capabilities of Vision Transformers to improve the accuracy and robustness of deepfake detection in video content.
The architectures rely on the EfficientNet-Pytorch and ViT-Pytorch libraries for their foundational components. The repository is licensed under the MIT License, ensuring permissive use and modification.
How It Works
The workflow for implementing and utilizing these architectures involves several key steps: setup, data preprocessing, model training, and evaluation. Each phase is designed to ensure compatibility with the specified datasets and to optimize performance for deepfake detection tasks.
Setup
To begin, users must clone the repository and configure the Python environment using Conda.
Data Preprocessing
Preprocessing is a critical phase in the workflow, as it involves extracting and organizing facial data from video datasets. The repository leverages Selim Seferbekov’s implementation for face detection and extraction. Users must first detect faces within videos using the detectfaces.py script, which employs the MTCNN (Multi-task Cascaded Convolutional Networks) algorithm.
If face detection fails for certain videos, users can adjust the threshold values in the facedetector.py file to improve detection rates. After detecting faces, the extractcrops.py script extracts the facial regions as individual images, organizing them into a structured directory hierarchy. This hierarchy separates data into training, validation, and test sets, with subfolders for different deepfake manipulation techniques (e.g., Deepfakes, Face2Face, FaceSwap) and original content.
The dataset structure must adhere to a specific format to ensure compatibility with the training and evaluation scripts. For example, the training set should include subfolders for each video, with frames labeled sequentially (e.g., 00.png, 10.png).
Training and Evaluation
Training the models requires downloading pre-trained weights for either the Efficient ViT or Cross Efficient ViT architectures.
The training process for both architectures follows a standardized workflow.
Evaluation is conducted using the test.py script, which requires the path to a pre-trained model and a configuration file. By default, the script evaluates performance on the DFDC dataset, but users can specify alternative datasets or adjust parameters such as the number of frames per video and batch size.
Dataset Auditing and Provenance
One of the key use cases for this tool is dataset auditing, where organizations can verify the integrity and provenance of video datasets. By applying the Efficient ViT or Cross Efficient ViT models, users can systematically identify deepfakes within large-scale video collections, ensuring that training data for other AI models remains authentic and unbiased.
Benchmarking and Comparative Analysis
Researchers can use this implementation to benchmark their own deepfake detection models against the proposed architectures. The repository provides a standardized framework for training and evaluation, allowing for fair comparisons across different methodologies.
Hybrid Architecture Design
The core innovation of this work lies in the hybrid architecture that integrates EfficientNet with Vision Transformers.
Support for Multiple Datasets
The repository supports two widely used datasets in deepfake research: DFDC and FaceForensics++. DFDC focuses on high-quality deepfakes generated using advanced techniques, while FaceForensics++ includes a variety of manipulation methods, such as Face2Face, FaceSwap, and NeuralTextures.
Customizable Training and Evaluation Parameters
Users have extensive control over the training and evaluation processes through configurable parameters.
Pre-Trained Models and Reproducibility
The repository provides pre-trained models for both Efficient ViT and Cross Efficient ViT, reducing the time and computational resources required for initial deployment. These models have been trained on the DFDC dataset, achieving competitive performance metrics.
Integration with Existing Libraries
The implementation leverages established libraries such as EfficientNet-Pytorch and ViT-Pytorch, which are widely adopted in the machine learning community.