How Judgeval Works
Judgeval is an open-source Python SDK designed to enhance the reliability of LLM-powered agents by providing tracing and evaluation capabilities. It captures agent runs, applies custom or built-in scorers, and converts production failures into structured regression datasets. This tool integrates with a hosted Judgment platform, enabling teams to detect failures, analyze root causes, and validate fixes using real production data before deployment.
At its core, Judgeval operates on two primary mechanisms: OpenTelemetry-based tracing and agent judges. OpenTelemetry integration allows developers to instrument any function with @Tracer.observe(), automatically capturing inputs, outputs, and LLM token usage. This compatibility ensures seamless integration with existing observability stacks. Agent judges, on the other hand, define prompt-based scorers to evaluate agent behaviors at scale. These judges produce structured outputs—scored and labeled—that describe agent actions, accumulating into a searchable record of behavior over time.
Judgeval supports both live production traffic and historical trace replay, enabling teams to validate fixes against real-world scenarios. Online monitoring capabilities allow for server-side scoring of live traffic without latency impact, surfacing detected behaviors as structured signals. These signals can be configured to trigger alerts in platforms like Slack, ensuring regressions and recurrences are promptly addressed.
The tool also emphasizes broad integrations, offering auto-instrumentation for major LLM providers such as OpenAI, Anthropic, Google GenAI, and Together AI. Framework support extends to LangGraph, OpenLit, and Claude Agent SDK, ensuring compatibility with diverse agent development environments. Additionally, Judgeval provides a CLI for managing agents, traces, judges, and evaluations directly from the terminal, as well as an MCP Server for connecting to AI tools compatible with the MCP protocol.
Judgeval’s architecture is built around the concept of a continuous-improvement loop for agents. When failures occur, the tool helps teams trace the execution path, identify patterns, and measure the impact of issues. This evidence-based approach ensures that fixes are validated against production data, reducing the risk of introducing new problems. The system also detects recurring behaviors and alerts teams to potential regressions, maintaining agent reliability over time.
To facilitate querying and analysis, Judgeval introduces JQL (Judgment Query Language), a domain-specific language for filtering and inspecting traces. Offline traces and spans can also be queried, enabling evaluation of test runs against production data. This flexibility ensures that teams can investigate issues comprehensively, whether they originate from live traffic or controlled environments.
The tool’s design addresses common challenges in agent development, such as missed tool calls, stale context, or faulty reasoning, by preserving the entire execution path as evidence. This approach enables teams to search across traces, identify related failures, and pinpoint root causes with precision. By automating the evaluation process, Judgeval reduces the manual effort required to monitor and improve agent performance, making it a critical component for teams focused on production reliability.
Judgeval’s integration with the Judgment platform extends its functionality by providing a centralized interface for managing agent behaviors, deploying judges, and running evaluations. The platform’s ability to surface failure modes and usage patterns from unlabeled production trajectories further enhances its utility, offering insights that would otherwise require extensive manual analysis.
Use Cases
Judgeval is primarily used for testing and red teaming of LLM-powered agents, providing a structured approach to evaluating agent behavior and identifying vulnerabilities. One of its key applications is in detecting and triaging production failures. When an agent misbehaves—such as issuing incorrect refunds, failing to escalate cases appropriately, or providing inaccurate responses—Judgeval captures the entire execution trace. This trace includes inputs, outputs, and intermediate steps, allowing teams to analyze the root cause of the failure.
For example, in a scenario where an agent incorrectly processes a refund request, Judgeval can trace the execution path to determine whether the issue stems from a missed tool call, a timeout in policy lookup, or an incorrect interpretation of user input. By identifying the specific behavior that led to the failure—such as issuing credit before policy verification—teams can quantify the impact, such as the number of affected sessions or the financial loss incurred. This data-driven approach ensures that fixes are targeted and validated against real-world cases.
Another critical use case is the validation of proposed fixes before deployment. Judgeval allows teams to replay historical traces against a candidate agent, simulating production conditions to verify that the fix resolves the issue without introducing new problems. This capability is particularly valuable in enterprise environments, where agent failures can have significant financial or operational consequences. By testing fixes against a dataset derived from production failures, teams can ensure that the agent behaves as expected in diverse scenarios.
Judgeval also excels in monitoring agent behavior over time to detect regressions or model drift. As agents interact with users and evolve, their behavior may deviate from intended specifications. Judgeval’s online monitoring feature automatically scores live production traffic, surfacing structured signals that indicate deviations from expected behavior. These signals can be configured to trigger alerts, ensuring that teams are notified immediately when an agent begins to misbehave. This proactive approach minimizes the risk of undetected failures affecting end users.
In addition to failure detection, Judgeval supports the construction and refinement of evaluation rubrics based on verifiable signals from production data. Teams can define custom judges to assess specific aspects of agent performance, such as adherence to policies, accuracy of responses, or efficiency of tool usage. These judges generate structured outputs that accumulate into a searchable record, enabling teams to track behavior patterns and measure improvements over time.
Judgeval’s CLI and MCP Server integrations further expand its utility by enabling teams to manage agents and traces directly from their development environments. The CLI allows for terminal-based queries of trace history, deployment of judges, and inspection of detected behaviors, streamlining the evaluation workflow. The MCP Server extends this functionality by enabling interaction with AI assistants or IDEs, allowing developers to search traces, investigate behaviors, and run tests without leaving their preferred tools. This integration supports a more collaborative and efficient development process.
Key Features & Capabilities
Judgeval’s OpenTelemetry-based tracing is a foundational feature that enables comprehensive instrumentation of agent functions. By decorating functions with @Tracer.observe(), developers can automatically capture inputs, outputs, and LLM token usage, creating detailed traces of agent executions. These traces are compatible with existing observability stacks, ensuring that teams can leverage their current monitoring infrastructure. The granularity of the captured data allows for precise analysis of agent behavior, from individual tool calls to entire execution paths.
Agent judges are another core capability, providing a framework for defining prompt-based scorers to evaluate agent behaviors at scale. These judges produce structured outputs that include scores and labels describing how the agent acted during a given run. The accumulated data forms a searchable record of agent behavior, enabling teams to identify patterns, measure the frequency of specific behaviors, and assess their impact. Judges can be applied to both live production traffic and historical traces, ensuring that evaluations are consistent and comprehensive.
Online monitoring is a key feature that enables real-time scoring of production traffic without introducing latency. Detected behaviors are surfaced as structured signals, which can be configured to trigger alerts in platforms like Slack. This capability ensures that regressions and recurrences are promptly identified and addressed, reducing the time between failure detection and resolution. By automating the monitoring process, Judgeval minimizes the manual effort required to maintain agent reliability in production environments.
Judgeval supports a wide range of integrations with major LLM providers and agent frameworks, including OpenAI, Anthropic, Google GenAI, Together AI, LangGraph, OpenLit, and Claude Agent SDK. Auto-instrumentation simplifies the integration process, allowing developers to quickly enable tracing and evaluation for their existing agents.
The JQL (Judgment Query Language) is a powerful feature for querying and analyzing traces. JQL allows developers to filter traces by trace IDs, session IDs, or other criteria, enabling precise investigations into agent behavior. Queries can be scoped to offline traces, supporting evaluation of test runs against production data. This capability is particularly useful for validating fixes or identifying patterns in historical data. JQL’s syntax is designed to be intuitive, with support for functions like present() and discover() to refine search results.
Judgeval’s CLI provides a command-line interface for managing agents, traces, judges, and evaluations. Users can query trace history, deploy judges, inspect detected behaviors, and run evaluations directly from the terminal. The CLI’s integration with the Judgment platform ensures that all operations are consistent with the broader evaluation ecosystem, maintaining data integrity and usability.
The MCP Server is an advanced feature that connects Judgeval to any MCP-compatible AI tool. This integration allows developers to query agent traces, invoke judges, and browse detected behaviors directly within their AI assistant or IDE. By surfacing failures and behaviors in familiar environments, the MCP Server streamlines the debugging and evaluation process.
Judgeval’s ability to turn production failures into regression datasets is a unique capability that enhances its value for continuous improvement. By capturing and structuring failure cases, teams can create datasets that reflect real-world agent behavior. These datasets can be used to test proposed fixes, ensuring that they address the root cause of the issue without introducing new problems. This data-driven approach to validation reduces the risk of deploying flawed fixes, improving overall agent reliability.
These features enhance JQL’s flexibility, allowing developers to narrow queries more precisely and evaluate offline traces alongside production data.