AI Observability and Evaluation Platform for LLM Applications
Arize Phoenix is an open-source platform designed to provide observability, evaluation, and debugging capabilities for large language model (LLM) applications. Developed by Arize AI, Phoenix integrates with OpenTelemetry and OpenInference to offer a comprehensive workflow for tracing, evaluating, and iterating on AI systems. It is available under the Elastic License 2.0 and can be deployed locally, containerized, or in the cloud, making it adaptable to various environments.
Phoenix is part of Arize’s broader suite of tools, complementing Arize AX, a managed enterprise platform that shares the same underlying instrumentation standards. While Phoenix is self-hosted and open-source, Arize AX provides a managed alternative for production workflows requiring additional support and scalability.
How It Works
Phoenix operates by capturing detailed traces of LLM application executions, enabling users to analyze runtime behavior, debug issues, and optimize performance. The platform leverages OpenTelemetry for instrumentation, allowing it to integrate seamlessly with popular frameworks and LLM providers. Key components include tracing, evaluation, dataset management, experiment tracking, and prompt optimization.
Tracing with OpenTelemetry
Phoenix uses OpenTelemetry to trace LLM applications, capturing model calls, retrieval operations, tool usage, and custom logic. These traces provide a step-by-step view of application execution, helping users identify bottlenecks, debug failures, and understand latency sources. The platform supports auto-instrumentation for frameworks such as LlamaIndex, LangChain, DSPy, and Mastra, as well as LLM providers like OpenAI, Anthropic, Google GenAI, and AWS Bedrock. Integration is further extended through the OpenInference project, which standardizes instrumentation across multiple tools and providers.
Evaluation and Debugging
Evaluations in Phoenix allow users to measure the quality of LLM outputs using LLM-based evaluators, code-based checks, or human annotations. These evaluations can be attached to traces, spans, or datasets, providing consistent performance tracking and failure identification. The platform supports pre-built evaluators for tasks such as RAG relevance and answer relevance, as well as custom evaluators from tools like Ragas, Deepeval, and Cleanlab.
Dataset and Experiment Management
Phoenix enables users to create versioned datasets for experimentation, evaluation, and fine-tuning. These datasets can be grouped into experiments, allowing systematic testing of changes across different versions of prompts, models, or retrieval systems. The platform’s experiment tracking features facilitate A/B testing and performance comparison, ensuring that improvements are validated before deployment. Datasets can also be exported for fine-tuning or used in the Prompt Playground for iterative optimization.
Prompt Optimization and Playground
The Prompt Playground in Phoenix provides an interactive environment for experimenting with prompts, comparing models, and adjusting parameters. Users can replay traced LLM calls with different inputs to observe how changes affect outputs. Prompt versioning and tagging are supported, allowing systematic testing of variants across datasets. The Playground also integrates with Span Replay, enabling users to debug specific LLM calls by replaying them with modified inputs.
PXI: AI Engineering Agent
It is particularly useful for coding agents like Claude Code, Cursor, and Codex, which can interact with Phoenix through its remote MCP server.
Remote MCP Server
Phoenix features a built-in remote MCP server that allows MCP-compatible clients such as Claude Code, Cursor, and VS Code to query traces, datasets, experiments, and prompts directly. The MCP server supports OAuth authentication and dynamic client registration, requiring no pre-configuration. Clients authenticate via browser login or API keys, with tokens scoped to user permissions. The server exposes a compact code-mode surface, enabling agents to discover and compose operations through five primary tools: search, tags, get_schema, execute, and Monty-based Python execution for advanced filtering and aggregation.
Use Cases
Phoenix is tailored for AI engineers, data scientists, and developers working with LLM applications.
Debugging and Troubleshooting
By capturing detailed traces of LLM application executions, Phoenix helps users identify and resolve issues such as incorrect outputs, latency spikes, or retrieval failures. The platform’s tracing capabilities allow for granular analysis of model behavior, tool usage, and custom logic, making it easier to pinpoint root causes of failures. Evaluations and human annotations further validate the correctness of outputs, ensuring that debugging efforts are evidence-based.
Performance Optimization
Phoenix enables users to optimize LLM applications by comparing different versions of prompts, models, and retrieval systems. Experiment tracking allows for systematic A/B testing, where changes are validated against consistent datasets before deployment. The Prompt Playground facilitates iterative optimization, while dataset evaluators provide automated test cases for performance monitoring. These features help reduce trial-and-error cycles and improve the efficiency of AI development.
Experimentation and Iteration
The platform supports large-scale experimentation by allowing users to version datasets, track changes, and compare evaluation results across different configurations. This is particularly useful for fine-tuning models, testing new retrieval strategies, or iterating on prompt designs.
Integration with Coding Agents
Phoenix’s remote MCP server and CLI tools (e.g., px setup) enable seamless integration with coding agents, allowing developers to interact with traces, datasets, and experiments without leaving their development environment. Agents can query Phoenix for recent traces, debug failures, inspect experiments, and manage prompts, reducing context-switching and accelerating workflows. This integration is especially valuable for teams using tools like Claude Code, Cursor, or VS Code for AI development.
Key Features and Capabilities
Phoenix offers a range of features designed to enhance AI observability, evaluation, and debugging. Below are its core capabilities, supported by its open-source architecture and OpenTelemetry integration.
OpenTelemetry and OpenInference Integration
- Supports auto-instrumentation for frameworks: LlamaIndex, LangChain, DSPy, Mastra, Vercel AI SDK, LangGraph, CrewAI, and others.
- Compatible with LLM providers: OpenAI, Anthropic, Google GenAI, AWS Bedrock, OpenRouter, LiteLLM, and more.
- Extends instrumentation through the OpenInference project, ensuring compatibility with emerging tools and providers.
- Provides lightweight Python and TypeScript sub-packages for tracing, evaluation, and client interactions.
Evaluation Tools
- LLM-based evaluators for response and retrieval quality assessment.
- Code-based checks for deterministic validation of outputs.
- Human annotation support for ground-truth labeling in the UI.
- Integration with third-party evaluators like Ragas, Deepeval, and Cleanlab.
- Dataset evaluators for automated performance tracking during experiments.
Dataset and Experiment Management
- Versioned datasets for experimentation, evaluation, and fine-tuning.
- Experiment tracking to compare different configurations systematically.
- Span replay for debugging specific LLM calls with modified inputs.
Prompt Optimization
- Prompt Playground for interactive experimentation with prompts and models.
- Side-by-side comparison of prompt variants and model performances.
- Integration with coding agents for automated prompt testing and iteration.
Deployment Flexibility
- Runnable locally via pip install arize-phoenix or uvx arize-phoenix serve.
- Containerized deployment options via Docker Hub or Kubernetes (Helm chart).
- Self-hosted or managed (Arize AX) deployment options with shared instrumentation standards.
Security and Privacy
- OAuth2-based authentication for the remote MCP server, with dynamic client registration.
- API key support for headless environments or clients without browser login capabilities.
- Sandboxed code execution in Monty for secure agent-written Python operations.
- Environment variables for configuring MCP server behavior, including code mode and skill paths.
Community and Support
- Active GitHub repository with release notes, documentation, and issue tracking.
- Comprehensive documentation covering installation, deployment, tracing, and integrations.