How τ-bench Works
τ-bench is a benchmark designed to evaluate the interaction between tool-using agents and simulated users in realistic customer-service scenarios. It focuses on two primary domains: retail and airline customer support. The benchmark simulates dynamic conversations where a language agent, equipped with domain-specific API tools and policy guidelines, interacts with users represented by language models. The goal is to assess whether agents adhere to policies and consistently complete tasks across varied user inputs.
The benchmark operates by defining structured tasks within the retail and airline domains. Users are simulated using language models, which generate responses based on predefined strategies. These strategies include tool-calling, ReAct (Reasoning + Acting), and other interaction patterns. The agent’s performance is measured by its ability to navigate these conversations while adhering to policies and achieving task completion.
These variants increase in complexity, testing the agent’s reliability under varying conditions. The benchmark tracks whether agents complete these tasks correctly and efficiently, providing a quantitative measure of their reliability.
The benchmark supports multiple agent strategies, including tool-calling (TC), ReAct, and Act (Acting). Each strategy is tested against different language models, such as GPT-4o, Claude 3.5 Sonnet, and Mistral Large.
User simulators in τ-bench are configurable, allowing researchers to test agents against various interaction patterns. Default configurations use GPT-4o as the user simulator with an LLM strategy, but alternatives include ReAct, reflection-based, and verification-based strategies. This flexibility ensures that agents are tested under diverse and realistic conditions, mimicking the unpredictability of human users.
To run τ-bench, users clone the repository and install it from source using pip. Required API keys for OpenAI, Anthropic, Google, Mistral, or AnyScale must be set as environment variables. Execution involves specifying the agent strategy, environment (retail or airline), model, and user simulator. For example, running a tool-calling agent in the retail environment with GPT-4o as both the agent and user model can be done via a command-line interface. Users can also target specific tasks by their IDs, enabling focused evaluations.
The benchmark includes an auto error identification tool that automates the detection and classification of faults in agent-user interactions. This tool assigns responsibility for errors to either the user, agent, or environment and categorizes them into types such as partially completed goals, incorrect tool usage, or unintended actions. While this feature relies on LLMs and may produce inaccuracies, it significantly reduces the manual effort required to analyze long and complex interaction trajectories.
Historical trajectories for both retail and airline environments are provided in the repository to facilitate cost-effective evaluations. Contributions of new trajectories are encouraged through pull requests, fostering a collaborative improvement of the benchmark’s dataset.
Use Cases
τ-bench is primarily intended for teams evaluating the reliability of language agents in customer-service domains. Its structured tasks and simulated user interactions provide a rigorous testing ground for assessing whether agents can handle real-world scenarios consistently.
The benchmark is useful for red teaming exercises, where agents are tested under adversarial or edge-case conditions to identify vulnerabilities. By simulating users with varying strategies, researchers can uncover weaknesses in agent behavior, such as policy violations or failure to handle ambiguous requests. This proactive approach helps improve agent robustness before deployment in production environments.
Researchers and developers can use τ-bench to compare the performance of different language models and agent strategies. The leaderboard format provides clear metrics for evaluating models like GPT-4o, Claude 3.5 Sonnet, and Mistral Large across multiple task variants.
The inclusion of auto error identification and historical trajectories makes τ-bench a practical tool for debugging and optimizing agent performance. Teams can analyze failed interactions to pinpoint recurring issues, such as incorrect tool usage or incomplete task execution. The ability to classify errors by type and responsibility streamlines the debugging process, enabling targeted improvements to agent logic or policy guidelines.
Students and practitioners can explore how agents respond to user requests, the impact of different strategies, and the challenges of maintaining consistency in dynamic conversations. The benchmark’s transparency and reproducibility make it an ideal tool for academic research and industry training programs.
Key Features & Capabilities
- Realistic Domains: τ-bench focuses on retail and airline customer-service scenarios, which are representative of common AI deployment environments.
- Simulated User Interactions: User simulators are implemented using language models with configurable strategies, including tool-calling, ReAct, and reflection-based approaches.
- Policy Compliance Measurement: The benchmark explicitly measures whether agents adhere to predefined policies during task execution.
- Multi-Model and Multi-Strategy Support: τ-bench supports a wide range of language models and agent strategies, including tool-calling, ReAct, and Act. This flexibility allows researchers to benchmark different approaches and models against a standardized set of tasks, facilitating fair comparisons.
- Auto Error Identification: The built-in error identification tool automates the analysis of interaction trajectories, classifying faults by type and responsibility. This feature reduces the manual effort required to diagnose issues, making it easier to iterate on agent improvements.
- Historical Trajectories: Pre-recorded interaction datasets for retail and airline environments are provided, enabling cost-effective evaluations without additional API usage. This resource is particularly useful for initial testing or educational purposes.
- Extensibility and Collaboration: The benchmark is designed to be extensible, with support for new domains, strategies, and error classification methods. Contributions from the research community are encouraged, ensuring continuous improvement and expansion of the benchmark’s capabilities.