LiteLLM: An Open Source AI Gateway for Unified LLM Access
LiteLLM is an open source AI gateway developed by BerriAI, designed to simplify access to over 100 large language models (LLMs) from providers such as OpenAI, Anthropic, Azure, and AWS Bedrock. It operates as both a Python SDK and a proxy server, offering a unified interface that abstracts away provider-specific differences in authentication, request formats, and error handling. By standardizing interactions through the OpenAI API format, LiteLLM enables seamless integration with existing applications without requiring code modifications.
The project was initiated in 2023 and is actively maintained, with recent updates including Docker image signature verification via cosign, ensuring the integrity of releases. LiteLLM is licensed under a custom or multiple open source licenses, as specified in its repository.
How It Works
LiteLLM functions through two primary components: the Python SDK and the AI Gateway (Proxy Server). The Python SDK allows developers to directly integrate LiteLLM into their applications, while the proxy server acts as a centralized service for managing LLM access across teams or organizations.
Python SDK
The Python SDK provides a drop-in replacement for the OpenAI client library. Developers can use familiar methods such as completion(), embedding(), and image_generation() to interact with any supported LLM. The SDK handles provider-specific authentication, request transformations, and error mapping to OpenAI-compatible exceptions, ensuring consistency across all integrations.
Example usage for calling OpenAI and Anthropic models demonstrates this uniformity:
response = completion(model="openai/gpt-4o", messages=[{"role": "user", "content": "Hello!"}])
The SDK also supports advanced features such as streaming responses, retry logic, and fallback mechanisms for high availability.
AI Gateway (Proxy Server)
The LiteLLM proxy server acts as an intermediary between client applications and LLM providers. It centralizes API keys, enforces rate limits, tracks budgets, and applies guardrails such as content filtering and PII masking. The proxy supports virtual keys for fine-grained access control, allowing organizations to manage permissions and spending at the team or user level.
To deploy the proxy, users configure a config.yaml file and run a Docker container. The proxy exposes an OpenAI-compatible endpoint, meaning any client library designed for OpenAI will work with LiteLLM without modification.
litellm --model gpt-4o
client = openai.OpenAI(apikey="anything", baseurl="http://0.0.0.0:4000")
response = client.chat.completions.create(model="gpt-4o", messages=[{"role": "user", "content": "Hello!"}])
The proxy includes an admin dashboard for monitoring usage, costs, and system health, along with integration points for observability tools like Langfuse, MLflow, and Helicone.
Use Cases
LiteLLM is designed to address the complexity of managing multiple LLM providers, which often require distinct SDKs, authentication methods, and request formats.
- Unified Access for Developers: Developers can switch between providers without rewriting code, as all interactions adhere to the OpenAI format. This is particularly useful for prototyping and testing across different models.
- Enterprise-Ready Gateway: Organizations can deploy LiteLLM as a centralized service to manage LLM access, budgets, and security policies. Features like virtual keys, spend tracking, and guardrails make it suitable for production environments.
- Agent and Tool Integration: LiteLLM supports Agent-to-Agent (A2A) protocols and Model Control Protocol (MCP) tools, allowing developers to integrate custom agents and tools into their workflows. For example, agents built with LangGraph or Azure AI Foundry can be invoked through the LiteLLM gateway.
- Cost and Usage Tracking: The proxy provides granular visibility into API usage, enabling teams to monitor spending per key, team, or user. This is critical for optimizing budgets and preventing cost overruns.
Key Features and Capabilities
LiteLLM’s feature set is designed to address the operational challenges of multi-provider LLM integration.
- Supported Endpoints: LiteLLM supports a comprehensive range of OpenAI-compatible endpoints, including /chat/completions, /messages, /embeddings, /images, /audio, /batches, /rerank, and /moderations.
- Provider Agnosticism: The gateway abstracts provider-specific quirks, such as authentication patterns and request formats. For instance, it translates between OpenAI’s /chat/completions and Anthropic’s /messages endpoints, ensuring consistent behavior.
- This consistency reduces debugging overhead.
- Observability and Logging: LiteLLM integrates with observability tools to log input/output data, track latency, and monitor costs. Supported tools include Langfuse, MLflow, Helicone, Lunary, and Promptlayer. Custom callbacks can also be implemented for additional logging needs.
- Security and Compliance: The proxy includes features such as SSO/SAML integration, audit logging, and content filtering to ensure compliance with organizational security policies. Guardrails can be configured to block sensitive data or enforce usage policies.
- Multi-Tenancy and Access Control: Virtual keys allow organizations to segment access by team or user, with independent budgets and rate limits. The admin dashboard provides oversight into usage patterns and spending trends.
- The proxy also supports load balancing and automatic fallbacks to maintain availability during provider outages.
- Agent and Tool Support: LiteLLM natively supports Agent-to-Agent (A2A) protocols and Model Control Protocol (MCP) tools. Developers can invoke agents built with frameworks like LangGraph or Azure AI Foundry directly through the gateway. For MCP tools, LiteLLM provides a centralized endpoint for managing access and permissions.
Supported Providers
LiteLLM currently supports over 100 LLM providers, including major platforms such as OpenAI, Anthropic, Azure, AWS Bedrock, and Google Gemini. The full list of supported providers and their compatible endpoints is documented in the official repository. Notable providers include:
- OpenAI (all endpoints)
- Anthropic (including Claude models)
- Azure (including Azure OpenAI and Azure Text)
- Cohere, Hugging Face, and Ollama
- Custom and third-party providers via the custom and customopenai configurations.
Recent Developments
- Docker image signature verification using cosign, ensuring the integrity of releases and protecting against tampering.
- Synchronization of pricing data for providers like Google Gemini, OpenAI, and AWS Bedrock to ensure accurate cost tracking.
- Enhancements to the A2A and MCP gateways, including support for dynamic OAuth configurations and improved error handling.
Technical Implementation
LiteLLM is implemented in Rust and Python, combining performance and ease of use. The Rust-based proxy server handles high-throughput requests efficiently, while the Python SDK provides a developer-friendly interface.
For advanced use cases, LiteLLM offers utilities such as /utils/transform_request, which allows developers to inspect and modify requests before they are sent to providers. This is particularly useful for debugging prompt formatting or provider-specific parameters.
The project’s documentation provides detailed guides on setup, configuration, and integration, including tutorials for Python SDK usage, proxy deployment, and agent/tool integration.