Video Face Manipulation Detection Through Ensemble of CNNs
This repository implements a system for detecting video-based face manipulation using an ensemble of Convolutional Neural Networks (CNNs). The work was presented at the 25th International Conference on Pattern Recognition (ICPR) in 2020 and published in IEEE Xplore and arXiv. The project was developed by the Image and Sound Processing Lab (ISPL) at Politecnico di Milano, participating in the Kaggle Deepfake Detection Challenge where it achieved the 41st position out of 2116 teams, placing in the top 2% on the private leaderboard.
The implementation focuses on detecting manipulated facial videos, a critical task in ensuring data and training integrity in AI systems. The repository is actively maintained, with issues tracked for troubleshooting.
How It Works
The system relies on preprocessing video datasets to extract and index facial frames, followed by training and testing CNNs for manipulation detection. The pipeline involves several key steps: dataset preparation, model training, and evaluation.
Dataset Preparation
The repository supports three datasets: Facebook’s DeepFake Detection Challenge (DFDC) train dataset, FaceForensics++, and Celeb-DF (v2). While Celeb-DF was not used in the original paper, indexing scripts are provided for compatibility. The preprocessing script, makedataset.sh, indexes samples and extracts faces using 32 frames per video, a parameter adjustable in extractfaces.py. For DFDC, only the training split is utilized.
The indexcelebdf.py script allows indexing Celeb-DF videos similarly to DFDC and FaceForensics++, enabling consistent pipeline execution. Users can then proceed with face extraction and further processing.
Model Training
The repository includes scripts for training both non-siamese and siamese architectures. Non-siamese models, such as EfficientNetB4 and EfficientNetB4Att, are trained directly using trainbinclass.py with the --net parameter. Siamese architectures, like EfficientNetB4ST and EfficientNetB4AttST, require a two-step process:
- First, train the architecture as a feature extractor using traintriplet.py, specifying the model name without the 'ST' suffix (e.g., EfficientNetB4).
- Second, fine-tune the model with trainbinclass.py, this time including the 'ST' suffix and providing the path to the pre-trained feature extractor weights via the --init argument.
The trainall.sh script consolidates all training commands, with detailed comments guiding usage.
Pretrained Models and Testing
Pretrained weights for all architectures are available via a Dropbox link, organized by network and dataset. Each directory contains bestval.pth, representing the best-performing weights based on validation set performance. Testing commands are compiled in testall.sh, and Jupyter notebooks in the notebook folder facilitate result computations.
For quick evaluation, users can test pre-trained models on custom videos or images using provided Jupyter notebooks:
- Image prediction notebook
Use Cases
This implementation is designed for researchers and practitioners working on deepfake detection, particularly in video-based face manipulation scenarios.
- Dataset Auditing: Verifying the integrity of video datasets used in training AI models to ensure they are free from manipulated content.
- Provenance Verification: Assessing the authenticity of facial videos in forensic investigations or media verification contexts.
- Benchmarking: Serving as a reference implementation for comparing new detection algorithms against established CNN-based ensembles.
The repository’s participation in the Kaggle Deepfake Detection Challenge demonstrates its effectiveness in competitive environments, making it a valuable resource for evaluating detection performance.
Key Features & Capabilities
The repository provides a comprehensive toolkit for video face manipulation detection, supported by the following features:
- Multi-Dataset Compatibility: Works with DFDC, FaceForensics++, and Celeb-DF (v2), ensuring broad applicability across different manipulation techniques.
- Ensemble Learning: Combines multiple CNN architectures, including EfficientNet and Xception, to improve detection accuracy through diverse feature extraction.
- Siamese and Non-Siamese Architectures: Supports both binary classification and triplet-loss-based feature extraction, catering to different training paradigms.
- Preprocessing Pipeline: Automates face extraction and indexing, reducing manual effort and ensuring consistency across datasets.
- Pretrained Models: Offers ready-to-use weights for immediate testing, accelerating research and development cycles.
- Reproducibility: Detailed scripts and documentation allow users to replicate experiments, ensuring transparency and reliability.
- Modular Design: Separates training, testing, and inference steps, enabling flexibility for customization and experimentation.
The repository’s reliance on GPL 3.0 licensing ensures open access while maintaining compliance with copyleft principles, encouraging collaboration and adaptation within the research community.