Detecting Deepfakes with Self-Blended Images: Methodology and Implementation
The Self-Blended Images (SBI) approach, introduced in the CVPR 2022 Oral paper Detecting Deepfakes with Self-Blended Images by Kaede Shiohara and Toshihiko Yamasaki, presents a novel method for detecting deepfake videos by generating self-blended images. The method was recognized for its effectiveness in cross-dataset evaluations, achieving high accuracy across multiple benchmarks including Celeb-DF-v2, DFD, DFDC, DFDCP, and FFIW.
By training models on these self-blended images, researchers aim to enhance the model’s ability to generalize and detect deepfakes even when they originate from unseen datasets or generation methods.
The implementation provided by the authors is built using PyTorch and is designed to be modular, allowing researchers to experiment with different datasets and configurations. The repository includes pretrained models, preprocessing scripts, and inference tools, making it accessible for both academic and commercial use under specific licensing terms.
The project was initially released in April 2022, with updates and bug fixes added subsequently. Notably, a bug in the preprocessing code was addressed in March 2023, ensuring reproducibility of experimental results.
Users are required to download datasets such as Celeb-DF-v2 or FaceForensics++ and place them in designated folders within the project directory. Pretrained models, specifically EfficientNet-B4 weights trained on SBI datasets (FF-raw and FF-c23), are also provided for direct inference.
How It Works
The SBI methodology operates through a series of well-defined steps, beginning with the extraction of video frames and facial landmarks. The preprocessing pipeline involves two key scripts: one for cropping frames using Dlib and another for refining bounding boxes with RetinaFace.
The resulting self-blended images are then used to train a deep learning model, specifically an EfficientNet-B4 architecture, which learns to distinguish between real and manipulated content.
The model’s performance is evaluated using cross-dataset testing, where it is challenged with deepfake videos from datasets it was not explicitly trained on.
Use Cases
The repository’s support for multiple datasets, including Celeb-DF-v2 and FaceForensics++, expands its applicability to various research scenarios. The inclusion of pretrained models also lowers the barrier to entry for new users, allowing them to quickly deploy the method without extensive training from scratch.
Key Features and Capabilities
The Self-Blended Images repository offers several key features that enhance its utility for deepfake detection research. Firstly, it provides pretrained models trained on self-blended images derived from the FF-raw and FF-c23 datasets. For example, the model trained on FF-raw achieves 93.82% accuracy on Celeb-DF-v2 and 97.87% on DFD, indicating strong cross-dataset performance.
Users can also optionally augment the preprocessing pipeline with additional tools, such as the Face-Xray library, to further improve model performance.
The inference capabilities of the repository are another notable feature.
Additionally, the repository maintains a clear and structured documentation, including a changelog that tracks updates and bug fixes. The licensing terms also provide clarity, with the code and models freely available for research purposes while requiring a license agreement for commercial use.
The repository also supports ongoing research through its integration with newer publications. For example, the authors have released a follow-up paper, ExposeAnyone, which builds on the SBI methodology to address personalized audio-to-expression diffusion models. This evolution highlights the project’s role in advancing the field of deepfake detection and its adaptability to emerging challenges.