Open Evaluator Language Models and the Prometheus-Eval Library
Prometheus-Eval is an open-source framework designed to evaluate language models (LLMs) in generation tasks. It provides open evaluator language models, specifically Prometheus 2, and a library for using them as LLM-as-a-judge systems. These models offer absolute or pairwise grading against custom rubrics, serving as an open alternative to proprietary judge models like GPT-4.
The project is developed by KAIST and collaborators, with the library released under the Apache 2.0 license, ensuring permissive use and modification. It is implemented in Python and is primarily used for testing and red teaming LLMs, providing a structured way to assess model performance against predefined criteria.
The framework supports both local inference through vLLM and remote inference via LLM APIs, facilitated by LiteLLM.
How It Works
Prometheus-Eval operates by integrating open evaluator models with a library that enables grading responses based on custom rubrics. The system supports two primary grading modes: absolute grading, which assigns a score from 1 to 5, and pairwise grading, which compares two responses to determine the better one.
The model then generates feedback and assigns a score based on how well the response meets the criteria.
Pairwise grading, on the other hand, compares two responses to the same instruction. The model assesses both responses against a rubric and determines which one is superior, providing detailed feedback for each. This method is useful for fine-tuning models using Direct Preference Optimization (DPO) datasets, where explicit preferences between responses are required.
It supports local deployment via vLLM and remote APIs through LiteLLM, making it accessible for both individual researchers and large-scale deployments.
Use Cases
Prometheus-Eval is primarily used for evaluating the quality of LLM-generated responses in various scenarios. One key application is in the creation of high-quality datasets for fine-tuning models.
Another use case is in Direct Preference Optimization (DPO), where Prometheus 2 evaluates pairs of responses to determine which one is preferred. This process generates DPO datasets that can be used to further refine model behavior, aligning outputs more closely with desired preferences. The framework has been successfully applied to distill datasets like UltraInteract_sft, demonstrating its effectiveness in improving dataset quality.
Prometheus-Eval is also employed in red teaming exercises, where models are tested for robustness, safety, and alignment.
These evaluations demonstrate that Prometheus 2 achieves high correlation with human judgments and proprietary models like GPT-4, making it a reliable tool for LLM assessment.
Key Features and Capabilities
The Prometheus 2 models, built on Mistral-7B and Mixtral-8x7B, are optimized for both absolute and pairwise grading.
The library provides pre-defined evaluation templates for common use cases, such as pairwise comparison, faithfulness, correctness, and relevance. These templates simplify the integration of Prometheus models into existing evaluation pipelines, allowing users to focus on defining custom rubrics rather than crafting evaluation prompts from scratch.
Prometheus-Eval also supports dynamic rubric creation, enabling users to define evaluation criteria tailored to specific tasks or domains.
Recent releases, such as v0.1.20, have included fixes for score parsing and dependency updates, ensuring stability and compatibility with the latest tools and libraries.
Model Variants and Performance
Prometheus-Eval offers multiple model variants to suit different computational constraints and performance requirements.
The integration of Prometheus-Eval with tools like distilabel further extends its capabilities, enabling automated dataset curation and preference learning. This combination allows researchers to build high-quality SFT and DPO datasets efficiently, reducing reliance on proprietary models for evaluation tasks.