MLCommons AI Safety Benchmark v0.5: First Benchmark Design

Research question

Can an industry consortium standardise safety benchmarking of chat systems?

Key findings

Taxonomy of 13 hazard categories and a graded scoring approach - Evolved into the AILuminate v1.0 benchmark in December 2024

Sources

arxiv.orghttps://arxiv.org/abs/2404.12241

Contribute

Spot an error, or know a record we should add? Members can propose additions and submit corrections.

Submit research Report a correction or request removal

Related in Research Paper Library

Dynamic Conformance Testing of WebGPU Through Specification-Driven Mutation

WebGPU is a low-level graphics and compute API that exposes modern GPU functionality to web applications. While the official WebGPU Conformance Test Suite (CTS) focuses on well-formed usage under the WebGPU…

Evaluation science2026

FASTAR: FRI Accelerator for Scalable Transparent ARguments of Knowledge

Zero-Knowledge Proofs (ZKPs) enable a prover to cryptographically convince a verifier of the validity of a statement without revealing any underlying secrets, forming a foundational primitive for verifiable…

Evaluation science2026

GuidedRay: Diversity-Guided Direction Discovery for Targeted Hard-Label Black-Box Attacks

Deep neural networks are vulnerable to adversarial attacks. Among black-box attacks, targeted decision-based attacks are particularly difficult: the attacker observes only the target model's top-1 label and aims to…

Evaluation science2026

Introducing v0.5 of the AI Safety Benchmark from MLCommons

Research question

Key findings

Sources

Contribute

Related in Research Paper Library

Dynamic Conformance Testing of WebGPU Through Specification-Driven Mutation

FASTAR: FRI Accelerator for Scalable Transparent ARguments of Knowledge

GuidedRay: Diversity-Guided Direction Discovery for Targeted Hard-Label Black-Box Attacks