Harbor: Framework for Evaluating and Improving Agents in Containerized Environments
Harbor is a framework developed by the creators of Terminal-Bench, designed for evaluating and optimizing agents and language models in containerized environments. It provides a standardized approach to benchmarking, allowing teams to assess and compare coding and terminal agents at scale. Harbor supports reproducible task sandboxes, enabling consistent and reliable evaluation across different environments.
The framework is built to accommodate a wide range of agents, including Claude Code, OpenHands, Codex CLI, and others. It also facilitates the creation and sharing of custom benchmarks and environments, making it adaptable to diverse research and development needs. Harbor integrates with multiple cloud providers such as Daytona, Modal, LangSmith, Blaxel, Novita Sandbox, Tensorlake, and Runta, enabling parallel execution of experiments across thousands of environments.
The framework’s design ensures that experiments can be replicated with minimal variability, improving the reliability of results.
How It Works
Harbor operates by defining tasks in a structured format, which can be executed in isolated containerized environments. These environments simulate real-world scenarios, such as coding tasks or terminal interactions, providing a controlled setting for agent evaluation. Users can specify agents, models, and datasets through command-line interfaces, allowing for flexible configuration of experiments.
The framework supports concurrent execution of tasks, enabling efficient benchmarking across multiple agents and models. For example, running Terminal-Bench 2.0 with an agent like Claude Code and a model such as Anthropic’s Claude Opus 4.1 can be initiated with a single command.
Harbor’s architecture includes integration with various cloud providers, which allows users to scale evaluations to thousands of environments simultaneously. This parallel execution capability is particularly useful for large-scale experiments, where performance and resource management are critical.
Installation of Harbor is straightforward, with support for both uv and pip package managers. Once installed, users can run evaluations locally or deploy them to cloud environments by specifying the appropriate provider.
Use Cases
Harbor is primarily used by teams developing or comparing coding and terminal agents. Researchers and developers can leverage the framework to evaluate the performance of their agents against established benchmarks or custom tasks.
The framework is also valuable for red teaming exercises, where agents are tested for robustness, security, and adherence to safety protocols. This makes it a useful tool for identifying vulnerabilities or unintended behaviors in agents.
Harbor’s compatibility with multiple cloud providers enables distributed evaluations, which are essential for large-scale experiments. Teams can run thousands of tasks in parallel, significantly reducing the time required for benchmarking.
Key Features and Capabilities
One of its core capabilities is the support for structured task definitions, which ensure reproducibility and consistency across evaluations. Tasks are defined in a format that includes metadata such as versioning, dependencies, and execution parameters, allowing for precise control over experimental conditions.
The framework integrates with a variety of agents and models, including those from major providers like Anthropic, OpenAI, and custom implementations. This flexibility allows users to evaluate a wide range of agents, from proprietary solutions to open-source alternatives. Harbor also supports custom agents, enabling developers to test their own implementations within the framework’s sandboxed environments.
Parallel execution is a defining feature of Harbor, enabling users to run evaluations across thousands of environments simultaneously. Users can specify the number of concurrent tasks, optimizing resource usage based on their requirements.
For example, version 0.23.0 included fixes for background persistence, structured capability declarations for agents, and improvements to trajectory criteria handling.
Harbor’s documentation includes guides for custom agents, remote rollouts, and job results viewing, further enhancing its accessibility.
These safety features are critical for maintaining the integrity of evaluations, particularly in red teaming scenarios.
The framework’s support for reinforcement learning rollouts allows developers to collect trajectory data, which can be used to optimize agent policies. This capability is integrated with tools like LangSmith and Daytona, enabling seamless data collection and analysis.
Harbor’s documentation and community resources, such as the Harbor Cookbook, provide end-to-end examples and guides for users. The framework’s permissive Apache 2.0 license encourages collaboration and open-source contributions, fostering a growing ecosystem of tools and integrations.