Deepfake Video Detection Using Generative Convolutional Vision Transformer (GenConViT)
GenConViT is a deepfake detection framework designed to address the growing threat of manipulated video content. Developed by researchers Deressa Wodajo, Hannes Mareen, Peter Lambert, Solomon Atnafu, Zahid Akhtar, and Glenn Van Wallendael, the model integrates convolutional neural networks and vision transformers to enhance detection accuracy. The framework combines ConvNeXt and Swin Transformer architectures for feature extraction while employing Autoencoder (ed) and Variational Autoencoder (vae) modules to analyze latent data distributions. This dual-network approach enables GenConViT to identify subtle visual artifacts and inconsistencies in deepfake videos, improving generalizability across diverse datasets.
Additional funding came from the Flemish government’s COM-PRESS project, IDLab at Ghent University—imec, Flanders Innovation and Entrepreneurship (VLAIO), and the European Union. The project reflects a collaborative effort to combat disinformation by advancing AI-driven media integrity solutions.
The model’s effectiveness has been validated through rigorous testing on four benchmark datasets: DFDC (Deepfake Detection Challenge), FF++ (FaceForensics++), DeepfakeTIMIT, and Celeb-DF v2. GenConViT achieved an average accuracy of 95.8% and an AUC (Area Under the Curve) value of 99.3%, demonstrating strong performance in distinguishing between authentic and manipulated content.
How It Works
GenConViT operates through a hybrid architecture that merges convolutional and transformer-based feature extraction with generative modeling. The framework consists of two primary networks: one based on an Autoencoder (ed) and the other on a Variational Autoencoder (vae). Both networks are trained to reconstruct input data while capturing latent representations that expose inconsistencies in deepfake videos.
The ConvNeXt-Swin Hybrid layer serves as the backbone for feature extraction, combining the strengths of convolutional operations (ConvNeXt) and self-attention mechanisms (Swin Transformer). The Autoencoder and Variational Autoencoder modules then analyze the extracted features to detect anomalies in the latent space, which are often indicative of manipulation.
During training, the model processes video frames organized into three directories: train, valid, and test, each containing subfolders for fake and real samples. The training script supports customizable parameters, including the choice between Autoencoder (ed) or Variational Autoencoder (vae) variants, the number of epochs, batch size, and the option to resume training from a pretrained model. Metrics and weights are saved in a designated weight folder for later use.
For inference, GenConViT accepts video inputs or preprocessed datasets (DFDC, FaceForensics, DeepfakeTIMIT, or Celeb-DF) and extracts a specified number of frames (default: 15) for analysis. The prediction script supports half-precision (float16) computation to optimize performance. Pretrained weights for both the Autoencoder and Variational Autoencoder variants are available on Hugging Face, reducing storage requirements by excluding unnecessary components (e.g., the VAE weights are ~2.6GB instead of 5–7GB).
Use Cases
GenConViT is primarily designed for applications requiring high-accuracy deepfake detection, particularly in scenarios where media integrity is critical.
- Dataset Auditing: Researchers and organizations can use GenConViT to verify the authenticity of video datasets, ensuring that training or evaluation data is free from manipulated content.
- Provenance Verification: Platforms hosting user-generated content, such as social media or news outlets, can deploy GenConViT to authenticate videos before publication. This helps mitigate the spread of misinformation and deepfake-related disinformation campaigns.
- Forensic Analysis: Law enforcement and investigative agencies can leverage GenConViT to analyze suspicious video evidence, identifying potential tampering or fabrication. The model’s high accuracy and generalizability make it suitable for forensic applications where precision is paramount.
The model’s compatibility with multiple datasets (DFDC, FaceForensics++, DeepfakeTIMIT, Celeb-DF v2) ensures broad applicability across different deepfake generation techniques, from facial manipulation to voice cloning.
Key Features & Capabilities
GenConViT distinguishes itself through several technical and performance-oriented features:
- Hybrid Architecture: The combination of ConvNeXt and Swin Transformer enables efficient extraction of both spatial and contextual features, improving detection robustness.
- Generative Modeling: The use of Autoencoder and Variational Autoencoder modules allows the model to learn latent representations of authentic and manipulated content.
- High Accuracy Metrics: GenConViT achieves an average accuracy of 95.8% and an AUC of 99.3% across tested datasets, outperforming many existing deepfake detection models. These metrics reflect its ability to generalize across diverse deepfake generation methods.
- Dataset Flexibility: The model supports multiple benchmark datasets, including DFDC, FaceForensics++, DeepfakeTIMIT, and Celeb-DF v2.
- Efficient Inference: Pretrained weights are optimized for storage and computational efficiency, with the VAE variant reducing file size to ~2.6GB. The prediction script also supports half-precision (float16) computation, accelerating inference on compatible hardware.
- Modular Training: Users can customize training parameters, such as model variant (ed or vae), epochs, batch size, and pretrained model initialization. This modularity facilitates experimentation and adaptation to specific requirements.
- Open-Source Availability: Released under the MIT License, GenConViT provides full access to its codebase, pretrained models, and documentation.
The implementation requires Python 3.x and dependencies such as PyTorch, torchvision, decord, dlib, opencv, and facerecognition.
Implementation Details
GenConViT’s codebase is structured to facilitate both training and inference workflows. Key components include:
- Training Script (train.py): Accepts command-line arguments for dataset path, model variant (ed or vae), epochs, batch size, and optional pretrained model loading. Example commands demonstrate usage with sample data and specific configurations.
- Prediction Script (prediction.py): Processes video inputs or datasets, extracting frames for analysis. Users can specify the model variant, dataset type, frame count, and precision settings. Pretrained weights for both variants are available via Hugging Face for direct download.
The repository also includes a BibTeX entry for citation, ensuring proper attribution to the original research paper published on arXiv (2307.07036) and MDPI.