OpenLLMetry: OpenTelemetry Instrumentation for LLM Applications
OpenLLMetry is an open-source project designed to provide observability for large language model (LLM) applications. Built on top of OpenTelemetry, it captures and exports detailed telemetry data, including prompts, completions, tool calls, and vector database operations. This data can then be sent to any OpenTelemetry-compatible backend, ensuring seamless integration with existing observability solutions. The project is maintained by Traceloop under the Apache 2.0 license and is available in Python, with a JavaScript/TypeScript version also available as OpenLLMetry-JS.
The semantic conventions developed for OpenLLMetry have been adopted into the OpenTelemetry project itself, reflecting its growing influence in the field of LLM observability.
OpenLLMetry is structured as a set of extensions that build upon OpenTelemetry’s existing instrumentation capabilities. It includes standard OpenTelemetry instrumentations for LLM providers and vector databases, alongside a proprietary Traceloop SDK. The SDK simplifies the setup process while still producing OpenTelemetry-compatible data, allowing users to connect to their preferred observability stack without vendor lock-in.
The project’s design ensures that users already instrumented with OpenTelemetry can incorporate OpenLLMetry’s extensions directly, minimizing disruption to existing workflows.
How It Works
OpenLLMetry operates by instrumenting key components of LLM applications, including interactions with LLM providers, vector databases, and frameworks. It captures telemetry data such as prompts, completions, tool calls, and API interactions, then exports this data in a standardized format compatible with OpenTelemetry backends. This approach allows for end-to-end tracing of LLM workflows, providing visibility into performance, latency, and errors.
The core functionality is accessible via the Traceloop SDK, which can be initialized with minimal setup. Users install the SDK via pip and initialize tracing with a single line of code, such as Traceloop.init(). For local development, users can disable batch sending to view traces in real time by setting disablebatch=True.
OpenLLMetry’s architecture supports both direct instrumentation of LLM providers and vector databases, as well as integration with popular frameworks like LangChain, LlamaIndex, and CrewAI.
The project’s telemetry collection is limited to exception detection within instrumentations, ensuring that updates from LLM providers do not disrupt functionality. Telemetry data is anonymous and does not include personally identifiable information. Users can review the collected data in the project’s Privacy documentation. Telemetry is only collected when using the SDK; direct instrumentation does not trigger telemetry collection.
Use Cases
OpenLLMetry is primarily used for monitoring and debugging LLM applications, offering insights into the performance, efficiency, and reliability of AI-driven workflows. By capturing detailed traces of prompts, completions, and tool interactions, it enables developers to identify bottlenecks, latency issues, and errors in real time.
One key use case is tracing interactions with LLM providers such as OpenAI, Anthropic, Mistral AI, and Google’s Gemini. OpenLLMetry instruments these providers, allowing users to monitor API calls, response times, and token usage.
Vector database operations are another critical area of focus. OpenLLMetry supports instrumentation for databases like Chroma, Pinecone, Qdrant, and Weaviate, enabling users to track embedding generation, similarity searches, and data retrieval processes. This is essential for applications involving retrieval-augmented generation (RAG), where vector database performance directly affects output quality.
These integrations allow developers to trace workflows across entire applications, from prompt engineering to post-processing. For example, a LangChain-based chatbot can be fully instrumented to monitor each step, including data loading, LLM inference, and response generation.
OpenLLMetry’s compatibility with OpenTelemetry ensures that collected data can be exported to a wide range of backends, including Datadog, Honeycomb, New Relic, and Google Cloud. This interoperability allows organizations to leverage their existing observability infrastructure, reducing the need for additional tooling.
Key Features & Capabilities
OpenLLMetry’s primary feature is its comprehensive instrumentation of LLM providers and vector databases. It supports a broad range of services, including OpenAI, Anthropic, Cohere, Mistral AI, and AWS Bedrock, as well as vector databases like Chroma, Pinecone, and Weaviate. This extensive coverage ensures that users can monitor interactions with virtually any modern LLM or vector storage solution.
The project’s Traceloop SDK simplifies the instrumentation process. Users can initialize tracing with minimal configuration, and the SDK automatically captures relevant telemetry data.
OpenLLMetry adheres to OpenTelemetry standards, ensuring that generated traces are compatible with any OpenTelemetry-compatible backend. This compatibility extends to popular observability platforms like Datadog, Honeycomb, and Grafana, as well as cloud-based solutions such as Google Cloud, Azure Application Insights, and AWS X-Ray.
The project maintains a strong focus on community engagement and contributions. Users can contribute through code, documentation, or discussions via GitHub Issues, Slack, or Twitter.
OpenLLMetry’s semantic conventions have been adopted into the OpenTelemetry project, ensuring that its standards for LLM observability are recognized and implemented across the broader observability ecosystem.