How It Works
OpenInference is designed as a complementary framework to OpenTelemetry, focusing on tracing AI applications with a particular emphasis on large language models (LLMs) and their surrounding ecosystems. It operates by defining semantic conventions that standardize how LLM invocations, tool usage, retrieval processes, and agent interactions are recorded. These conventions are documented in markdown files within the spec directory and are agnostic to transport protocols or file formats, ensuring compatibility with JSON, ProtoBuf, and DataFrames.
The framework provides instrumentation libraries for popular machine learning SDKs and frameworks, enabling developers to trace LLM calls, tool integrations, and retrieval operations. These instrumentations are structured to capture detailed context about AI application workflows, including interactions with vector stores, external APIs, and search engines. By adhering to OpenTelemetry standards, OpenInference ensures that traces generated can be processed by any OpenTelemetry-compatible backend, including Arize Phoenix and Arize AX.
The semantic conventions define standardized attributes and metadata for tracing AI operations, ensuring consistency across different tools and frameworks. The instrumentation libraries, available in Python and JavaScript, provide reusable utilities, decorators, and configuration helpers to simplify the implementation of tracing in AI applications. These libraries are tailored to support a wide range of SDKs, including OpenAI, LangChain, LlamaIndex, and others.
Additionally, OpenInference introduces span processors to normalize and convert data from other instrumentation libraries, unifying traces for consistent analysis. This feature is particularly useful for integrating traces from tools like OpenLIT and OpenLLMetry (Traceloop), ensuring that data from disparate sources can be aggregated and analyzed seamlessly. The framework also supports manual tracing through annotation-based methods, such as ByteBuddy for Java, and typed span classes for more granular control over trace generation.
The specification and instrumentation are actively maintained, with recent updates including the addition of new LLM providers like Oracle as a well-known value for llm.provider and enhancements to support specific SDKs, such as the inclusion of a FinishReason attribute for LangChain4j.
OpenInference is transport-agnostic, meaning it can be integrated with various OpenTelemetry collectors and exporters. This flexibility allows developers to route traces to different backends, such as Arize Phoenix or Arize AX, depending on their infrastructure requirements.
Use Cases
OpenInference is primarily used to enhance observability in AI applications, particularly those involving LLMs, agents, and retrieval-augmented generation (RAG) systems. By providing detailed traces of LLM invocations, tool calls, and agent interactions, it enables developers to monitor performance, debug issues, and optimize workflows. This is critical for applications where AI models are integrated into production systems, as it allows for real-time visibility into how models are being used and where potential bottlenecks or errors may occur.
One key use case is in the development and deployment of AI agents, where multiple tools, APIs, and data sources are often orchestrated to achieve complex tasks. OpenInference’s instrumentation for frameworks like LangChain, LlamaIndex, and AutoGen AgentChat allows developers to trace the entire workflow of an agent, including how it retrieves information, interacts with external services, and generates responses. This level of detail is essential for identifying inefficiencies, such as slow retrieval queries or failed API calls, and for ensuring that agents adhere to expected behavior.
Another significant use case is in the monitoring of RAG systems, where LLMs are augmented with external knowledge bases or vector stores.
OpenInference is also used to standardize observability across multi-vendor AI ecosystems. For example, instrumentation libraries for AWS Bedrock, Google GenAI, and Anthropic SDKs allow developers to trace interactions with these services in a consistent manner, regardless of the underlying provider. This standardization simplifies the process of aggregating and analyzing traces from diverse sources, making it easier to compare performance across different models and services.
By capturing detailed traces of LLM calls and tool interactions, developers can identify and diagnose issues such as incorrect API responses, malformed prompts, or unexpected model behavior.
Key Features & Capabilities
OpenInference’s core feature is its semantic conventions, which define standardized attributes for tracing AI operations. These conventions cover a broad range of AI workflows, including LLM invocations, tool calls, retrieval operations, and agent interactions. By adhering to these conventions, developers can ensure that traces generated by different tools and frameworks are consistent and comparable. This standardization is achieved through attributes such as llm.provider, llm.model, tool.name, and retriever.query, which provide context about the components involved in an AI workflow.
The instrumentation libraries provided by OpenInference are designed to simplify the implementation of tracing in AI applications. For Python, these libraries include openinference-semantic-conventions, which defines the standardized attributes, and openinference-instrumentation, which offers reusable utilities, decorators, and configuration helpers. Additional libraries are tailored to specific frameworks, such as openinference-instrumentation-langchain for LangChain, openinference-instrumentation-llama-index for LlamaIndex, and openinference-instrumentation-openai for the OpenAI SDK.
OpenInference supports a wide array of AI frameworks and SDKs, including those for LLM providers like OpenAI, Anthropic, MistralAI, and AWS Bedrock, as well as agent frameworks like AutoGen, LangChain, and CrewAI. The framework also extends to JavaScript environments, with support for libraries like LangChain.js and Vercel AI SDK, making it accessible to developers across different programming ecosystems.
Span processors are another key capability of OpenInference, enabling the normalization and conversion of trace data from other instrumentation libraries. For example, the openinference-instrumentation-openlit processor converts traces from OpenLIT into a format compatible with OpenInference’s semantic conventions, while the openinference-instrumentation-openllmetry processor handles traces from OpenLLMetry (Traceloop). This functionality ensures that data from disparate sources can be aggregated and analyzed cohesively, reducing the complexity of managing multiple tracing systems.
OpenInference also includes features for manual tracing, which provide developers with fine-grained control over how traces are generated. In Java, this is supported through annotation-based tracing with ByteBuddy and typed span classes like LLMSpan and AgentSpan. Similarly, the Go implementation of OpenInference includes utilities for suppression, context attribute propagation, and trace configuration masking, enhancing security and flexibility in trace collection.
Recent updates to OpenInference have expanded its capabilities, particularly in supporting new LLM providers and enhancing trace attributes. Similarly, the inclusion of a FinishReason attribute for LangChain4j improves the granularity of traces for tool calls, providing clearer insights into how and why operations conclude. These updates demonstrate OpenInference’s ongoing evolution to meet the needs of an increasingly complex AI landscape.
OpenInference’s compatibility with OpenTelemetry ensures that traces generated by the framework can be processed by any OpenTelemetry-compatible backend. This flexibility allows developers to choose the observability platform that best fits their infrastructure, whether it is Arize Phoenix, Arize AX, or other OpenTelemetry collectors. This combination of flexibility and security makes OpenInference a versatile tool for AI observability across different environments.