OpenAI Evals: A Framework for Evaluating LLMs and LLM Systems
OpenAI Evals is a framework designed to evaluate large language models (LLMs) and systems built using LLMs. It provides an open registry of benchmark evaluations, including model-graded evals for tasks that lack a single correct answer. The framework is specifically tailored for teams developing custom evaluations against OpenAI-compatible endpoints, enabling rigorous testing and red teaming of AI systems.
The framework supports both predefined benchmarks and custom evaluations, allowing users to assess model performance across diverse dimensions. It is particularly useful for understanding how different model versions impact specific use cases, reducing the time and effort required for manual testing. OpenAI emphasizes that creating high-quality evaluations is one of the most impactful actions teams can take when building with LLMs.
OpenAI Evals operates under the MIT license, ensuring permissive use and modification by the community. The framework integrates with OpenAI’s API, enabling seamless execution of evaluations while accounting for associated costs. Additionally, users can log evaluation results to external databases like Snowflake, provided they configure the necessary environment variables.
The evals registry is managed using Git-LFS, allowing users to fetch specific evaluations or the entire dataset.
How It Works
OpenAI Evals follows a structured process for creating, running, and managing evaluations. The workflow begins with setting up an OpenAI API key, which is specified via the OPENAI_API_KEY environment variable. Users can also integrate with tools like Weights & Biases for additional functionality, though the primary dependency remains the OpenAI API.
To create custom evaluations, users are encouraged to clone the OpenAI Evals repository directly from GitHub. This approach allows for immediate reflection of changes without reinstallation, streamlining the development process. The repository includes pre-commit hooks for code formatting, ensuring consistency and adherence to best practices. Users can install these hooks by running pip install -e .[formatters] followed by pre-commit install, which automates formatting checks on every commit.
Running evaluations locally is straightforward once the package is installed via pip install evals. Detailed instructions for executing existing evaluations are provided in run-evals.md, while templates for creating new evals are outlined in eval-templates.md. For advanced use cases, such as evaluating prompt chains or tool-using agents, OpenAI Evals supports the Completion Function Protocol, offering flexibility in testing complex workflows.
Evaluations are defined using a combination of YAML and JSON files. For example, an evaluation designed to classify IT support tickets into categories like Hardware, Software, or Other would require a YAML file defining the grading criteria and a JSON file containing sample tickets with their correct labels.
When running an evaluation, users must define two key components: data_source_config and testing_criteria. The data_source_config outlines the schema for test data, including fields like ticket_text and correct_label, while testing_criteria specifies how model outputs are graded. For instance, a string_check grader can be used to verify if a model’s output matches the expected label exactly. Template syntax, such as {{ item.correct_label }}, dynamically inserts values from the test data into the evaluation process.
Test data is typically provided as a JSONL file, where each line represents a test case with input and ground truth labels. Users upload this file to the OpenAI platform, either through the dashboard or via API, and reference its unique ID during evaluation runs. OpenAI also provides examples of evaluations implemented in multiple ways, such as the CoQA dataset, to illustrate different evaluation templates and their applications.
Use Cases
OpenAI Evals is designed to address a wide range of evaluation needs, from basic model testing to complex system assessments. One primary use case is red teaming, where evaluations are used to identify vulnerabilities or unintended behaviors in AI systems.
Another critical application is performance benchmarking across different model versions. As AI models evolve, teams need a reliable way to compare how updates affect their specific use cases. For example, a team developing a customer support chatbot can use evaluations to track whether new model versions improve response accuracy or introduce regressions in handling specific ticket types.
The framework also supports the creation of private evaluations, allowing teams to assess model performance on proprietary or sensitive data without exposing the data publicly. By using private evaluations, teams can ensure that their models adhere to internal standards while maintaining confidentiality.
Users can follow existing templates to define evaluations using YAML and JSON files, eliminating the need for complex programming. This accessibility lowers the barrier to entry, enabling non-technical stakeholders to contribute to evaluation design.
OpenAI actively reviews contributions to the evals registry, particularly those that could inform improvements to future models. While custom code submissions are currently not accepted, model-graded evaluations defined in YAML are encouraged. This ensures that the registry remains focused on reusable, standardized evaluations while still capturing diverse use cases.
Key Features and Capabilities
One standout capability is the support for model-graded evaluations, which are essential for tasks without objective ground truth. For example, evaluating creative writing or open-ended questions requires subjective grading, where another model or human reviewer assesses the quality of responses.
The framework also integrates with OpenAI’s Dashboard, allowing users to configure and run evaluations directly within the interface.
OpenAI Evals provides comprehensive documentation and examples to guide users through the evaluation process. The build-eval.md guide walks users through creating a new evaluation from scratch, while custom-eval.md demonstrates how to implement custom evaluation logic. The examples folder contains end-to-end examples, and Jupyter notebooks offer interactive tutorials for hands-on learning.
Another key feature is the Completion Function Protocol, which supports advanced use cases such as evaluating prompt chains or tool-using agents. This protocol allows users to define complex evaluation workflows, including multi-step interactions or tool integrations, ensuring that evaluations accurately reflect real-world system behavior.
By using Git-LFS, the framework efficiently manages large evaluation datasets, reducing storage and download times. Additionally, the framework provides clear guidance on API costs, helping teams budget for evaluation runs and avoid unexpected expenses.
The disclaimer associated with OpenAI Evals clarifies the licensing terms and data usage policies. Contributions to the repository are subject to the MIT license, and users must ensure they have the rights to upload any data included in their evaluations. OpenAI reserves the right to use contributed data for improving its products, aligning with its broader mission to advance AI capabilities.