Implementation of Improved DeepFake Detection Using Whisper Features
This repository provides the codebase for the research paper titled Improved DeepFake Detection Using Whisper Features, published in 2023. The paper introduces a method for enhancing deepfake audio detection by incorporating features extracted from OpenAI’s Whisper model.
The project is maintained by Piotr Kawa and affiliated researchers, with the source code licensed under the MIT License, ensuring permissive use for academic and commercial applications. The primary focus is on ensuring data and training integrity in audio deepfake detection systems, particularly in auditing datasets and verifying provenance.
The implementation builds upon prior work in the field, including the Attack Agnostic Dataset repository, while introducing novel modifications to accommodate Whisper features. It is structured to support both training and evaluation pipelines, with configurations tailored for different deepfake detection models.
How It Works
The core mechanism of this implementation involves extracting features from audio samples using Whisper, a robust automatic speech recognition model. These features are then combined with traditional front-end processing techniques such as LFCC (Linear Frequency Cepstral Coefficients) or MFCC (Mel-Frequency Cepstral Coefficients) before being fed into deepfake detection models. This hybrid approach leverages Whisper’s ability to capture semantic and contextual information in speech, which is often overlooked in conventional deepfake detection methods.
The workflow begins with the preparation of datasets, specifically the ASVspoof2021 DeepFake (DF) subset and the In-The-Wild dataset. These datasets are structured to include both genuine and synthetic speech samples, allowing the models to learn distinguishing patterns. The repository provides scripts to download and preprocess these datasets, ensuring compatibility with the training and evaluation pipelines.
These embeddings are then processed alongside traditional features to train or fine-tune deepfake detection models.
Each of these models can be configured to use Whisper features either independently or in combination with LFCC/MFCC front-end processing.
- RawNet3: rawnet3 (without Whisper integration)
The selection of the front-end processing method is specified in the configuration files, which are written in YAML format. These files define parameters such as the model architecture, input channels, optimizer settings (e.g., learning rate and weight decay), and other hyperparameters critical to the training process.
Use Cases
This implementation is primarily targeted at researchers and practitioners working in the domain of audio deepfake detection.
- Dataset Auditing: Verifying the integrity and provenance of audio datasets used in deepfake detection research.
- Benchmarking and Comparison: Serving as a baseline or reference implementation for evaluating new deepfake detection techniques. The repository provides pretrained models and evaluation scripts, allowing researchers to compare their methods against established benchmarks.
The use of the ASVspoof2021 DF subset and the In-The-Wild dataset ensures that the implementation is tested on diverse and challenging datasets, including both controlled and real-world scenarios. This makes it a valuable tool for developing robust deepfake detection systems that can generalize across different types of synthetic speech.
Key Features and Capabilities
The repository includes several scripts and configurations to streamline the training, fine-tuning, and evaluation processes.
- downloadwhisper.py: Downloads the Whisper encoder required for feature extraction.
- trainandtest.py: Executes the full training and testing pipeline. This script accepts command-line arguments to specify dataset paths, configuration files, batch sizes, epochs, and other training parameters.
python trainandtest.py --asvpath ../datasets/deepfakes/ASVspoof2021/DF --inthewildpath ../datasets/releaseinthewild --config configs/finetuning/whisperspecrnet.yaml --batchsize 8 --epochs 5 --trainamount 100000 --validamount 25000
- trainmodels.py: Dedicated script for training deepfake detection models. This allows users to train models independently of the evaluation process, providing flexibility in experimental setups.
- evaluatemodels.py: Script for evaluating the performance of trained models.
The repository also includes pretrained models corresponding to those reported in the paper. These models are available for direct use, allowing researchers to immediately apply the Whisper-enhanced detection methods without the need for training from scratch. The configurations for these models are stored in the configs/training/ directory, with example files provided for each supported architecture.
Dependencies for the repository are managed using conda, with specific versions of Python, PyTorch, and related libraries required for compatibility. The key dependencies include:
- Asteroid-Filterbanks 0.4.0
- OpenAI Whisper (commit 7858aa9c08d98f75575035ecd6481f462d66ca27)
These dependencies ensure that the implementation runs consistently across different environments, reducing potential issues related to version conflicts or compatibility problems.
The configuration files play a central role in defining the behavior of the training and evaluation pipelines. An example configuration file for training an LCNN model with LFCC features is provided below:
path: "trainedmodels/lcnn/ckpt.pth"
frontendalgorithm: ["lfcc"]
This structure allows users to customize the training process by adjusting hyperparameters, model architectures, and data processing steps.
The repository also acknowledges the Attack Agnostic Dataset repository as a foundational resource, incorporating its dependencies and extending its functionality to include Whisper-based feature extraction. This integration ensures that the implementation benefits from established practices in deepfake detection while introducing novel enhancements.