OpenLIT: OpenTelemetry-Native Observability for AI Agents and Applications
OpenLIT is an open-source observability and evaluation platform designed specifically for AI agents and applications. Built on OpenTelemetry, it provides comprehensive tracing, cost monitoring, GPU utilization tracking, prompt management, secrets handling, and built-in guardrails for AI systems.
The platform addresses the complexity of modern AI workflows, which extend beyond simple LLM calls to include tool invocations, retrieval-augmented generation (RAG), memory management, sub-agent interactions, prompt engineering, and even code modifications. By tracing every component—from individual LLM calls to entire agent sessions—OpenLIT offers end-to-end visibility into AI system behavior, enabling teams to debug, optimize, and govern their deployments effectively.
OpenLIT’s architecture is designed to fit within existing infrastructure. It listens for OpenTelemetry Protocol (OTLP) data on ports 4317 (gRPC) and 4318 (HTTP), allowing it to integrate with OpenTelemetry Collectors or directly receive telemetry from AI applications. This flexibility ensures that organizations can adopt OpenLIT without disrupting their current observability pipelines, while still benefiting from its specialized AI-focused features.
The platform is self-hosted by default, with support for Docker and Kubernetes deployments. This approach ensures that sensitive telemetry—such as prompts, API keys, and evaluation results—remains under the control of the deploying organization, aligning with privacy and compliance requirements. OpenLIT’s design prioritizes observability, measurability, and continuous improvement of AI systems, making it a tool for both development and production environments.
How It Works
OpenLIT operates by instrumenting AI applications and agents to generate OpenTelemetry traces and metrics. This process begins with the installation of OpenLIT’s SDK, which is available for Python and TypeScript. Once installed, the SDK automatically instruments supported LLM providers, frameworks, vector databases, and other AI infrastructure, exporting telemetry to an OpenTelemetry-compatible endpoint—typically the OpenLIT receiver running locally or within the organization’s infrastructure.
Users can start OpenLIT in under five minutes by cloning the repository, running a Docker Compose command, and configuring the OTLP endpoint. The SDK then initializes instrumentation with a single call, such as openlit.init(), which captures traces for LLM calls, tool invocations, prompts, and other AI workflow components. These traces are sent to OpenLIT’s dashboard, where they can be explored for debugging, cost analysis, and performance optimization.
For coding agents—such as those used in development environments like Claude Code, Cursor, and Codex—OpenLIT provides a dedicated CLI tool. This tool simplifies instrumentation by allowing users to install vendor-specific integrations (e.g., for Cursor or Claude Code) with a single command.
At its core, OpenLIT’s functionality revolves around four key pillars: tracing, cost observability, evaluations, and governance. Traces provide a granular view of AI workflows, including spans for LLM calls, tool interactions, and retrieval operations. Cost observability tracks token consumption, model pricing, and GPU utilization, offering insights into operational expenses.
OpenLIT’s integration with OpenTelemetry ensures that telemetry can flow through existing observability ecosystems. This includes support for backends like ClickHouse, Tempo, Loki, and Prometheus, as well as third-party tools such as Datadog, Grafana, and Honeycomb. The platform’s modular design allows organizations to extend its functionality through custom connectors, rule engines, and evaluation criteria, tailoring it to their specific needs.
Use Cases
OpenLIT is tailored to address the unique challenges of AI observability, particularly in environments where AI agents interact with multiple components—LLMs, tools, retrieval systems, and sub-agents. One primary use case is debugging production AI systems. By tracing entire agent conversations, OpenLIT helps teams identify root causes of failures, such as tool call errors, retrieval issues, or LLM hallucinations. This level of detail reduces mean time to resolution (MTTR) by providing context-rich error traces, including span relationships and execution paths.
AI applications often incur significant expenses due to token usage, model selection, and GPU utilization. OpenLIT tracks these costs at granular levels—per request, per model, and per provider—enabling teams to optimize spending without sacrificing performance. For example, the platform can identify expensive model usage patterns or inefficient token consumption, allowing organizations to implement cost-saving measures such as model switching or prompt optimization.
Prompt management is a third major use case, particularly for teams collaborating on AI applications. OpenLIT’s Prompt Hub treats prompts as versioned artifacts, allowing teams to edit, deploy, and roll back prompts without modifying application code. Every prompt version is linked to its corresponding traces and evaluations, ensuring traceability from development to production. This feature is especially valuable for A/B testing, where teams can compare model performance, cost, and output quality across different prompt variants before deployment.
For coding agents, OpenLIT provides visibility into developer productivity and AI tooling efficiency. It tracks metrics such as token usage, lines of code added or removed, model acceptance rates, and even the number of commits or pull requests generated per user or team. This data helps organizations measure the impact of AI coding tools on engineering workflows, identify cost drivers, and ensure equitable access to resources across teams. Privacy controls, such as secret redaction and cohort-based anonymization, protect sensitive data while still enabling insightful analysis.
Evaluations are a cornerstone of OpenLIT’s functionality, enabling teams to assess AI output quality in both development and production environments. The platform supports LLM-as-a-judge evaluations, which automatically score outputs for criteria such as correctness, relevance, and adherence to guidelines. These evaluations can be run in real-time on live traces or programmatically in CI/CD pipelines, ensuring consistency between development and production. The results are linked to specific prompt versions, models, and traces, providing a clear audit trail for quality assurance.
The platform includes features such as trace governance panels, API key management, and rule engines that enforce organizational policies. For example, OpenLIT can detect and flag issues like stuck agent loops, tool misuse, or cost anomalies, integrating findings with frameworks like NIST AI Risk Management Framework (RMF), the EU AI Act, and OWASP ASI. This ensures that AI systems align with regulatory and security requirements while maintaining operational efficiency.
Tracing and Debugging
OpenLIT’s tracing capabilities are built on OpenTelemetry, providing detailed visibility into AI workflows. Every LLM call, tool invocation, prompt, and agent step is instrumented as a span, capturing metadata such as input prompts, output tokens, latency, and errors. This granularity allows teams to reconstruct entire agent conversations, identify bottlenecks, and diagnose failures. For example, if an agent fails due to a tool call error, OpenLIT’s traces will show the exact sequence of events leading to the failure, including the prompt, LLM response, and subsequent tool interactions.
The platform also includes specialized debugging features, such as generation-health chips that classify LLM outputs as truncated, filtered, empty, or swapped. These classifications are backed by API endpoints and trace filtering capabilities, enabling teams to proactively monitor output quality. Additionally, OpenLIT detects stuck agent loops—repeated tool calls that waste tokens and increase costs—flagging them in traces with counts and cost details. This helps teams identify inefficiencies and optimize agent workflows.
Cost and GPU Monitoring
The platform tracks token consumption, model pricing, and provider-specific costs, offering dashboards that break down expenses by request, model, and environment. This level of detail is particularly useful for organizations using multiple LLM providers or fine-tuned models, as it allows them to compare costs and performance across different options. OpenLIT also supports custom pricing models, enabling teams to apply their own cost structures to traces.
GPU monitoring is another critical capability, especially for organizations running AI workloads on high-performance hardware. This data is displayed alongside LLM call latency and token consumption, providing a holistic view of system performance. The platform’s GPU collector supports multiple modes—such as all, light, serving, training, and deep—allowing teams to tailor monitoring to their specific workloads.
Prompt Management and Versioning
OpenLIT’s Prompt Hub centralizes prompt management, treating prompts as versioned artifacts that can be edited, deployed, and rolled back independently of application code. This separation of concerns simplifies collaboration among teams, as prompts can be updated and tested without requiring code changes. Every prompt version is linked to its corresponding traces and evaluations, ensuring traceability from development to production. For example, if a prompt change leads to a quality regression, teams can quickly identify the specific version and trace that caused the issue.
The platform also supports A/B testing of prompts and models, allowing teams to compare cost, latency, and output quality before deployment. OpenGround, OpenLIT’s model comparison tool, runs side-by-side evaluations to help teams select the optimal model for their use case. This feature is particularly valuable for organizations experimenting with multiple LLMs or fine-tuned variants, as it provides data-driven insights into performance trade-offs.
Evaluations and Governance
These evaluations can be run automatically on live traces in production or programmatically in CI/CD pipelines, ensuring consistency across environments. The results are linked to specific prompt versions, models, and traces, providing a clear audit trail for quality assurance.
Governance features in OpenLIT include trace filtering, rule engines, and compliance checks. For example, a rule could enforce cost evaluations for expensive models in production environments or retrieve specific prompts based on context. OpenLIT also integrates with governance frameworks like NIST AI RMF, the EU AI Act, and OWASP ASI, providing modular policy packs that align AI systems with regulatory requirements.
Integrations and Extensibility
OpenLIT supports over 70 integrations with LLM providers, vector databases, AI frameworks, and GPU infrastructure. The platform’s OpenTelemetry-native design ensures compatibility with existing observability ecosystems, including backends like ClickHouse, Tempo, Loki, and Prometheus, as well as third-party tools like Datadog and Grafana.
Extensibility is another strength of OpenLIT, with support for custom connectors, evaluation criteria, and rule engines. Teams can extend the platform’s functionality to meet their specific needs, such as adding new governance policies or integrating with proprietary AI tools. The OpenPlait connectors framework, introduced in version 2.0, further enhances this capability by providing a standardized way to manage and deploy connectors across the organization.
Recent Enhancements
This expansion broadens the platform’s compatibility with emerging AI infrastructure. The release also introduced Mem0, Claude, and Zep memory connectors, enabling integration with popular memory management systems and enhancing traceability for agent states.
Other notable additions in version 2.1.0 include generation-health chips for UI-based filtering and backend classification of LLM spans, stuck-agent loop detection with trace filtering, and API key authentication with organization and project scoping. These features improve debugging, cost management, and security, making OpenLIT more robust for production environments. Version 2.0.0 introduced the OpenPlait connectors framework, which streamlines the management of integrations and dependencies, while also adding tool misuse trace-analysis dimensions to improve governance and compliance.
Conclusion
OpenLIT provides a comprehensive solution for AI observability, combining tracing, cost monitoring, prompt management, evaluations, and governance into a single, self-hosted platform. Its OpenTelemetry-native design ensures compatibility with existing observability workflows, while its extensive feature set addresses the unique challenges of AI system development and production. By offering end-to-end visibility, OpenLIT enables teams to debug, optimize, and govern their AI applications effectively, ensuring reliability, cost efficiency, and compliance.