How LangWatch Works
LangWatch is an open-source platform designed to address the operational challenges of deploying large language models (LLMs) and AI agents in production environments. It integrates tracing, testing, evaluation, and governance into a unified system, enabling teams to monitor, debug, and optimize their AI applications. The platform operates on two primary models: self-hosted deployments using Apache 2.0 licensed core components or a hosted solution via LangWatch Cloud, which shares the same underlying codebase.
At its core, LangWatch employs OpenTelemetry-based tracing to capture every LLM call, tool invocation, and user interaction. These traces are stored as detailed spans with metadata, allowing teams to reconstruct entire conversation flows, identify performance bottlenecks, and analyze failures. The platform supports a wide range of integrations, including coding assistants like Claude Code, Codex, GitHub Copilot, and opencode, as well as frameworks such as LangChain, LangGraph, and Vercel AI SDK.
LangWatch’s architecture is built around four key areas: Observability, Agent Testing, Evaluations, and Prompt Management. Observability provides real-time visibility into LLM operations, while Agent Testing enables multi-turn simulations against AI agents to validate behavior under realistic conditions. Evaluations allow teams to run experiments on datasets or monitor production traffic, and Prompt Management ensures versioned prompts are loaded dynamically at runtime. These features are designed to work independently or in combination, depending on the team’s needs.
The platform also includes an AI Gateway, which serves as a unified endpoint for all LLM calls within an organization. This gateway supports virtual keys with budget controls, ensuring secure and cost-efficient usage of AI models. Additionally, LangWatch offers governance capabilities, including anomaly detection, compliance monitoring, and an inventory of all AI tools and their usage patterns across the organization.
LangWatch’s self-hosting option allows organizations to deploy the platform on their own infrastructure using Docker or Helm, ensuring data sovereignty and customization. The hosted version, LangWatch Cloud, provides a free tier for initial setup and experimentation, with enterprise features available under a commercial license. Security and compliance are prioritized, with the platform adhering to GDPR standards and holding ISO 27001 certification. Vulnerabilities can be reported directly to the LangWatch security team for prompt resolution.
The platform’s development is community-driven, with contributions managed through GitHub issues and a Discord channel for discussions. The TypeScript-based SDK and core components are open-source, while enterprise modules require a separate license. Recent updates, such as those in versions 1.16.0 and 1.17.0, have introduced features like Instant Evals for real-time judgment of LLM outputs, improved query capabilities via LangWatchQL, and enhanced onboarding workflows.
Use Cases
One of its primary use cases is agent testing and evaluation, where teams can simulate multi-turn conversations to validate agent behavior before deployment. This approach reduces the risk of production failures by identifying edge cases, policy violations, and adversarial scenarios early in the development cycle. For example, LangWatch can automatically generate test scenarios from plain-language specifications, allowing product managers to define requirements without coding. These scenarios are then executed against the agent, with results scored using LLMs, custom code, or predefined workflows.
Another key use case is cost optimization for coding assistants. LangWatch provides detailed metrics on token usage, session costs, and team-level expenditures for tools like Claude Code, Codex, and GitHub Copilot. The platform also supports virtual keys with predefined budgets, which can be distributed to employees or customers to control spending at a granular level.
For organizations deploying AI in production, LangWatch offers real-time observability and governance. The platform traces every LLM call, tool invocation, and user interaction, providing a searchable and filterable record of all AI activity. This capability is particularly valuable for teams handling sensitive operations, such as payments or compliance-heavy workflows, where transparency and auditability are critical. LangWatch’s governance features include anomaly detection, compliance monitoring, and role-based access controls, ensuring that AI tools are used in accordance with organizational policies.
LangWatch also excels in prompt management and versioning. Teams can store, version, and dynamically load prompts at runtime, ensuring consistency across deployments. Langy generates rubrics for evaluating agent responses, drafts prompt improvements, and even creates pull requests for fixes, streamlining the collaboration between product and development teams. This integration reduces the time required for testing and iteration, allowing teams to ship AI features with greater confidence and fewer surprises in production.
Beyond these core use cases, LangWatch supports evaluation of mixed-media outputs, including images and text, through a unified scoring surface. The platform also enables teams to compare outputs from different models, prompts, or versions side by side, facilitating data-driven decision-making.
Observability
LangWatch’s observability capabilities are built on OpenTelemetry-based tracing, which captures every interaction within an AI system. These traces include LLM calls, tool invocations, and user inputs, providing a detailed record of the entire conversation flow. The platform supports multiple visualization formats, such as waterfall diagrams, flame graphs, topology maps, and sequence diagrams, allowing teams to analyze performance bottlenecks and failures from different perspectives. For example, a flame graph can reveal which tool calls are consuming the most time, while a sequence diagram can clarify the order of operations in a complex workflow.
Traces are searchable and filterable in real time, with support for plain-language queries and advanced faceting. This functionality enables teams to quickly locate specific interactions, such as a failed tool call or an anomalous response, without sifting through logs manually.
Agent Testing
Agent testing in LangWatch is simulation-based, allowing teams to evaluate AI agents under realistic conditions. Simulations can be created from plain-language specifications using Scenario, a tool that automatically generates test cases based on user-defined requirements. These simulations support multi-turn conversations, concurrent testing across projects, and adversarial scenarios designed to probe for jailbreaks, policy violations, or unsafe tool calls. For instance, a team can simulate a user attempting to bypass security checks or manipulate an agent into performing unauthorized actions, ensuring that such vulnerabilities are identified before deployment.
The JudgeAgent feature evaluates agent responses by reading the entire trace, including tool calls and intermediate steps, to provide a verdict with reasoning. This approach ensures that evaluations are comprehensive and context-aware, rather than relying on isolated output analysis. Teams can score responses using LLMs, custom code, or workflows, and compare outputs side by side to select the best model, prompt, or version. Evaluations can be run offline during development or in real time against production traffic, capturing signals such as latency, cost, and user satisfaction.
Evaluations
LangWatch’s evaluation capabilities extend to both offline experiments and real-time production monitoring. Teams can run evaluations on datasets to benchmark models, prompts, or agent configurations, or monitor live traffic to detect anomalies or performance degradation. The platform supports scoring for text, images, and mixed-media outputs, ensuring that evaluations are comprehensive and applicable to multimodal AI systems. For example, a team developing a visual question-answering system can evaluate both the textual responses and the generated images against ground-truth data.
The platform also supports A/B testing, allowing teams to compare two outputs side by side and select the better option based on predefined criteria.
Prompt Management
Prompt management in LangWatch ensures that prompts are versioned, dynamically loaded at runtime, and accessible across the organization. Teams can store prompts in a central registry, track their usage, and roll back to previous versions if issues arise.
Langy, the AI-driven assistant, plays a key role in prompt management by drafting revisions based on evaluation results. For instance, if an evaluation reveals that a prompt is generating low-quality responses, Langy can suggest improvements, which developers can then review and deploy via the Prompt Registry. This workflow reduces the manual effort required for prompt optimization and ensures that improvements are implemented consistently across the organization. Additionally, LangWatch supports privacy controls for prompt usage, allowing teams to restrict access to sensitive prompts or track who is using them.
AI Gateway and Governance
The AI Gateway in LangWatch serves as a unified endpoint for all LLM calls within an organization, supporting providers such as OpenAI, Anthropic, Azure OpenAI, and Bedrock. This gateway enables teams to manage API keys centrally, route requests based on policies, and enforce budgets at the user or team level. Virtual keys with predefined limits can be distributed to employees or customers, ensuring that usage remains within allocated budgets. For example, a company can provide developers with virtual keys tied to their individual budgets, preventing overspending on API calls.
Governance in LangWatch includes an inventory of all AI tools and sources within the organization, along with anomaly detection and compliance monitoring. Teams can define rules to flag unusual activity, such as sudden spikes in API usage or unauthorized tool calls, and investigate potential security or policy violations. This centralized view helps teams maintain visibility and control over AI usage across the enterprise.
Security and compliance are prioritized in LangWatch, with the platform adhering to GDPR standards and holding ISO 27001 certification. Self-hosted deployments allow organizations to maintain data sovereignty, while LangWatch Cloud provides a hosted option with enterprise-grade security. Vulnerabilities can be reported directly to the LangWatch security team, ensuring rapid resolution. The platform’s open-core model allows teams to self-host the Apache 2.0 licensed core components, with enterprise features available under a commercial license.