Deepfake Detection Challenge: NTech-Lab’s Third-Place Solution
NTech-Lab’s submission for the Deepfake Detection Challenge achieved a private leaderboard score of 0.43452, securing third place in the competition. The solution relies on an ensemble of three EfficientNet-B7 models, each employing distinct approaches to frame processing and augmentation techniques. No external datasets were used beyond pre-trained weights, adhering strictly to the challenge’s constraints.
NTech-Lab addressed this by implementing a mixup technique on aligned real-fake face pairs, where real and fake samples were linearly combined using a Beta distribution (alpha=beta=0.5). This approach ensured that the model focused on facial features rather than background inconsistencies, as the interpolated samples retained nearly identical backgrounds.
The mixup technique introduced uncertainty in model predictions, which was mitigated during inference through a confidence-weighted averaging mechanism. Predictions closer to 0.5—indicative of low confidence—were assigned lower weights, effectively acting as an attention mechanism to filter out unreliable predictions from low-quality frames (e.g., blurry or profile views).
Model Architecture and Ensemble Design
The solution employs three EfficientNet-B7 models, each pre-trained using Noisy Student weights. EfficientNet-B7 was selected after empirical comparisons with ResNet, ResNeXt, and SE-ResNeXt, demonstrating superior performance for this task. The input image dimensions for all models were standardized at 224x192 pixels, accommodating the smaller face sizes prevalent in the dataset.
Two of the models operate on individual frames (frame-by-frame), differing primarily in the size of the face crop and the augmentation strategies applied during training. The third model processes sequences of seven frames, incorporating a 3D convolutional layer into each EfficientNet-B7 block to capture temporal dependencies. This sequence-based model outperformed its frame-by-frame counterparts by a marginal but meaningful degree.
Data Augmentation Strategies
To enhance generalization, the team applied a combination of augmentations: AutoAugment, Random Erasing, Random Crops, and Random Horizontal Flips. These were carefully synchronized for real-fake pairs to preserve alignment during mixup. For the sequence-based model, augmentations were applied uniformly across frames belonging to the same clip to maintain temporal consistency.
A novel augmentation technique involved simulating real-world video compression artifacts. Short clips (50 frames) were cropped around faces, saved as PNG sequences, and re-encoded on-the-fly during training using ffmpeg with randomized parameters for FPS (15–30), resolution scaling (0.25–1.0), and CRF (17–40). This approach mirrored the degradations observed in the test data, as documented in the DFDC Preview Dataset paper, and significantly improved the model’s robustness to compressed video inputs.
Training and Inference Pipeline
Training and preprocessing required approximately five days on an NVIDIA DGX-1 system equipped with eight Tesla V100 GPUs (32GB each), 512GB RAM, and a 6TB SSD.
During inference, predictions were generated for a subsample of frames, half of which were horizontally flipped to further diversify inputs. The final video-level prediction was computed by averaging individual frame predictions, with weights proportional to their confidence scores. This method ensured that unreliable predictions—common in low-quality frames—had minimal impact on the overall result.
Key Features and Capabilities
- Mixup on Aligned Real-Fake Pairs: Mitigates overfitting by training on interpolated samples, forcing the model to focus on facial artifacts rather than background inconsistencies. The Beta distribution (alpha=beta=0.5) ensures high-probability selection of pure real or fake samples, preserving class separation.
- Video Compression Augmentation: Simulates real-world video degradation by dynamically re-encoding clips with randomized compression parameters. This technique improved resistance to artifacts introduced by low FPS, reduced resolution, and poor encoding quality, as identified in the DFDC Preview Dataset.
- 3D Convolutional Sequence Model: Captures temporal dependencies by processing sequences of seven frames (sampled at 1/15-second intervals). The addition of 3D convolutions to EfficientNet-B7 blocks provided a slight but consistent performance boost over frame-by-frame models.
- Confidence-Weighted Ensemble: Combines predictions from three models, with weights inversely proportional to prediction uncertainty. This approach effectively filters out noisy predictions from low-quality frames, such as those with blur or profile views, enhancing overall accuracy.
- Hardware Optimization: Leveraged a high-performance DGX-1 system to handle the computationally intensive training process, including on-the-fly video compression augmentations and large-scale preprocessing of video data.
Use Cases
- Dataset Validation for AI Training: Auditing synthetic datasets used in machine learning pipelines to detect inconsistencies or adversarial manipulations that could compromise model performance.
- Real-Time Video Analysis: Adapting the sequence-based model for applications requiring temporal analysis, such as surveillance or live-stream monitoring for deepfake content.
The modular design of the solution—comprising frame-by-frame and sequence-based models—allows for flexibility in deployment, depending on the specific requirements of the use case. For example, frame-by-frame models may suffice for static image analysis, while sequence models are better suited for dynamic video content.