Holistic Evaluation of Language Models: HELM Framework

Research question

How should language models be evaluated across capabilities, robustness, fairness and toxicity together?

Key findings

HELM evaluates 30 models on 16 scenarios and 7 metrics including robustness and calibration - Reveals trade-offs invisible to single-metric leaderboards

Sources

arxiv.orghttps://arxiv.org/abs/2211.09110

Contribute

Spot an error, or know a record we should add? Members can propose additions and submit corrections.

Submit research Report a correction or request removal

Related in Research Paper Library

Dynamic Conformance Testing of WebGPU Through Specification-Driven Mutation

WebGPU is a low-level graphics and compute API that exposes modern GPU functionality to web applications. While the official WebGPU Conformance Test Suite (CTS) focuses on well-formed usage under the WebGPU…

Evaluation science2026

FASTAR: FRI Accelerator for Scalable Transparent ARguments of Knowledge

Zero-Knowledge Proofs (ZKPs) enable a prover to cryptographically convince a verifier of the validity of a statement without revealing any underlying secrets, forming a foundational primitive for verifiable…

Evaluation science2026

GuidedRay: Diversity-Guided Direction Discovery for Targeted Hard-Label Black-Box Attacks

Deep neural networks are vulnerable to adversarial attacks. Among black-box attacks, targeted decision-based attacks are particularly difficult: the attacker observes only the target model's top-1 label and aims to…

Evaluation science2026