Real-event-anchored benchmark for detecting AI-generated videos in crisis settings
RA-Bench is a benchmark designed to evaluate the effectiveness of AI-generated video detection systems in real-world crisis scenarios. Unlike traditional benchmarks, it focuses on videos that are anchored to real footage of socially consequential events, such as wars, disasters, or public emergencies. The benchmark pairs real crisis videos with AI-generated continuations derived from the same initial frames and prompts, creating a controlled yet realistic testing environment. This approach forces detection systems to analyze subtle differences rather than relying on obvious visual mismatches, simulating conditions where generated content closely mimics reality.
The dataset comprises 17,886 videos, including 1,830 real anchors and 16,056 generated videos produced by nine distinct generators—four open-source and five closed-source. These videos span 10 high-risk categories and 44 subcategories, covering a wide range of crisis scenarios. The inclusion of human-deceptive generated videos (633 clips judged as real by all five assigned reviewers) and sequential dissemination pipelines further tests the robustness of detection models under realistic conditions.
RA-Bench addresses a critical gap in existing benchmarks by evaluating detectors not just on isolated clips but on content that has undergone transformations typical of social media dissemination. The benchmark’s structure ensures that evaluations reflect real-world challenges, where generated content may appear indistinguishable from authentic footage.
The project involves 35 contributors from 19 affiliations, bringing together expertise in video generation, multimedia forensics, multimodal reasoning, and responsible AI. Lead authors include Shuo Liang, Yixing Ma, Pengfei Zhou, and corresponding authors such as Wei Wang, Yang You, and others. Their collaborative effort ensures a rigorous and multidisciplinary approach to benchmarking AI-generated video detection in high-stakes contexts.
How It Works
RA-Bench operates on a source-matched evaluation framework, where each generated video is paired with its corresponding real anchor. This design ensures that detectors are tested on content that shares the same initial context, making it harder to distinguish between real and generated videos based on superficial cues. The benchmark evaluates detectors using two primary output formats: continuous fake scores and discrete real/generated decisions. Metrics such as AUC (Area Under the Curve), TPR@5%FPR (True Positive Rate at 5% False Positive Rate), balanced accuracy (BAcc), Macro-F1, and FakeR are computed separately for each generation source before averaging to produce benchmark-level results.
The evaluation process is divided into three distinct tracks:
- This track assesses how well detectors perform across different generation sources, ensuring fairness by treating each source equally regardless of the number of clips.
- RA-Bench-HumanProof: Consists of 633 generated videos that deceived all five human reviewers during initial screening. This subset challenges detectors to identify highly realistic content that has already evaded human detection.
- RA-Bench-LastMile: Evaluates detector robustness under six sequential dissemination conditions, simulating the transformations videos undergo during social media sharing.
Users can evaluate predictions using the provided scripts, which support CSV or JSONL input formats. Two coverage modes are available: public-media, which evaluates only redistributable real anchors, and full, which includes URL-only real anchors requiring additional rights compliance.
The benchmark’s repository includes metadata files such as rabenchmain.csv, rabenchhumanproof.csv, and rabenchlastmile.csv, which define the evaluation sets. Additional files like realrightsrelease.csv document the rights and attribution requirements for real anchors, ensuring responsible use and compliance with source terms.
Use Cases
RA-Bench is primarily designed for testing and red-teaming AI-generated video detection systems, particularly in high-risk scenarios where misinformation can have severe consequences. Its structured approach allows researchers and developers to assess the limitations of current detectors and identify areas for improvement.
- Detector Evaluation: Researchers can test how well existing detection models perform on realistic AI-generated content, especially in crisis-related contexts. The benchmark’s focus on human-deceptive videos and sequential dissemination conditions provides insights into the vulnerabilities of current systems.
- Generator Analysis: By evaluating outputs from both open-source and closed-source generators, RA-Bench helps identify which models produce the most convincing deceptive content. This information can guide developers in improving detection algorithms or hardening generation models against forensic analysis.
- Social Media Dissemination Studies: The LastMile track simulates the real-world transformations videos undergo during sharing, making it possible to study how platform processing affects detectability. This is critical for understanding the resilience of detection systems in environments where content is repeatedly modified.
- Responsible AI Research: The benchmark supports studies on the ethical implications of AI-generated media, particularly in contexts where deepfakes or synthetic content could manipulate public perception. By providing a controlled yet realistic dataset, RA-Bench enables researchers to explore the societal risks of AI-generated videos and develop mitigation strategies.
The dataset’s inclusion of crisis-related footage also makes it valuable for organizations focused on media authenticity, such as fact-checking platforms, cybersecurity firms, and government agencies tasked with countering disinformation. Its rigorous methodology ensures that evaluations are both technically sound and aligned with real-world operational challenges.
Key Features & Capabilities
- Real-Anchor Pairing: Each generated video is derived from a real anchor using the same initial frame and prompt, ensuring that detectors must analyze subtle contextual differences rather than relying on obvious artifacts. This approach better mimics real-world scenarios where generated content is designed to blend seamlessly with authentic footage.
- Human-Deceptive Subset: The inclusion of 633 videos that fooled all five human reviewers provides a stringent test for detectors. This subset highlights the gap between human perception and machine detection, emphasizing the need for more sophisticated forensic techniques.
- Sequential Dissemination Pipeline: The LastMile track introduces six conditions that simulate the transformations videos undergo during social media sharing, including compression, platform-specific processing, and metadata stripping.
- Source-Generalization Evaluation: Detectors are evaluated separately for each generation source before averaging results, ensuring that no single generator disproportionately influences the benchmark’s outcome.
- Comprehensive Metrics: The benchmark supports multiple evaluation metrics, including AUC, TPR@5%FPR, balanced accuracy, Macro-F1, and FakeR. These metrics offer a nuanced view of detector performance, capturing both continuous scoring and discrete classification capabilities.
- Rights-Compliant Release: The dataset includes detailed metadata on rights and attribution for real anchors, with 1,319 clips redistributed as media and 511 represented by source URLs.
- Scalability and Reproducibility: The repository provides scripts for dataset download, verification, and evaluation, enabling researchers to replicate experiments and integrate RA-Bench into their workflows. The use of standardized formats (CSV, JSONL) and clear documentation further enhances usability.
RA-Bench’s focus on crisis-related content also sets it apart from general-purpose benchmarks. By anchoring evaluations in real-world events, the dataset provides a more targeted and relevant testing ground for detectors deployed in high-stakes environments, such as conflict zones, natural disasters, or public health emergencies.