Introduction
Agentgateway is an open-source proxy designed to address the connectivity challenges inherent in agentic AI systems. Developed as the first complete solution for agentic AI connectivity, it operates as a unified data plane that proxies traffic between agents, large language models (LLMs), and tools. The project was contributed by Solo.io to the Linux Foundation in 2025, where it operates under the Apache 2.0 license.
Agentgateway integrates with AI-native protocols such as the Model Context Protocol (MCP) and Agent-to-Agent (A2A) communication standards. It serves as a centralized gateway for HTTP, gRPC, and TCP traffic, offering a cohesive infrastructure for both traditional application traffic and AI-specific workflows. The project is positioned as a critical component for enterprises seeking to adopt agentic AI responsibly at scale, providing governance, visibility, and security for complex AI workflows.
The project has gained recognition within the AI community, particularly for its role in bridging gaps in agentic workflows. It aligns with emerging specifications like MCP and A2A, offering scalable infrastructure that supports secure and interoperable agent communication. Agentgateway is designed to address the evolving needs of AI infrastructure, ensuring that connectivity, security, and observability are prioritized in agentic systems.
It is hosted under the Linux Foundation, ensuring adherence to open-source best practices and fostering collaboration among developers, platform engineers, and AI enthusiasts. The project’s roadmap includes ongoing enhancements to its core features, with contributions from the community shaping its future development.
How It Works
Agentgateway functions as a proxy that intercepts and manages traffic between agents, LLMs, and external tools. It supports three primary communication protocols: Model Context Protocol (MCP), Agent-to-Agent (A2A), and direct LLM traffic. The proxy routes requests through a unified data plane, applying authentication, authorization, observability, and multiplexing to ensure secure and efficient communication.
The architecture of Agentgateway is modular, allowing it to handle diverse traffic types without requiring separate gateways. It provides a drop-in solution for integrating with major LLM providers such as OpenAI, Anthropic, Gemini, and Bedrock, as well as self-hosted models. Traffic is routed intelligently based on factors like GPU utilization, KV cache availability, LoRA adapters, and queue depth, ensuring optimal performance for inference tasks.
Authentication mechanisms include JWT, API keys, OAuth, and OIDC, with TLS and mTLS ensuring secure communication channels.
Observability is built into Agentgateway through OpenTelemetry integration, providing metrics, logs, and traces for monitoring traffic flow, latency, and costs. The system includes an admin UI that allows users to inspect tool calls, responses, and routing decisions in real time. This transparency is critical for debugging, auditing, and ensuring compliance with organizational policies.
Agentgateway can be deployed in standalone mode or integrated into Kubernetes environments using Helm charts and the Gateway API. The agctl CLI tool further simplifies administration by allowing users to interact with the proxy’s configuration and monitor its state dynamically.
Use Cases
Agentgateway is designed to address the connectivity and security challenges faced by organizations implementing agentic AI systems. Its primary use cases include routing traffic to major LLM providers, managing interactions between agents and tools via MCP, and facilitating secure agent-to-agent communication using A2A protocols.
For LLM providers, Agentgateway acts as a unified gateway that abstracts differences between APIs from OpenAI, Anthropic, Gemini, and Bedrock. It offers budget and spend controls, prompt enrichment, load balancing, and failover mechanisms, ensuring reliable and cost-effective access to LLMs.
In the context of MCP, Agentgateway enables agents to interact with tools and external data sources securely. Authentication is handled via OAuth, ensuring that only authorized agents can invoke tools. This capability is essential for enterprises looking to deploy agentic workflows that rely on external services and APIs.
Agent-to-agent communication is another key use case, where Agentgateway enables secure and observable interactions between autonomous agents. It supports capability discovery, modality negotiation, and task collaboration, allowing agents from different frameworks (e.g., LangChain, CrewAI, ADK) to communicate seamlessly. This interoperability is critical for building complex, distributed AI systems where agents must coordinate their actions.
Agentgateway also addresses observability and governance requirements for agentic AI deployments. Its built-in UI and OpenTelemetry integration provide real-time visibility into traffic patterns, policy enforcement, and cost tracking. Administrators can monitor token usage, latency, and routing decisions, ensuring that AI workflows remain within defined constraints. This level of transparency is particularly valuable for enterprises that must comply with regulatory or internal security policies.
Additionally, Agentgateway supports spend control mechanisms, allowing organizations to attribute, price, and cap AI-related costs. It provides hard limits on token consumption and dollar spend, ensuring that budgets are respected before invoices are generated. This feature is essential for managing the financial impact of AI workloads, particularly in environments where multiple teams or users share access to LLMs and tools.
Key Features & Capabilities
- LLM Gateway: Routes traffic to major LLM providers through a unified OpenAI-compatible API. Features include budget and spend controls, prompt enrichment, load balancing, and failover mechanisms.
- MCP Gateway: Facilitates secure communication between agents and tools using the Model Context Protocol. Enables agents to invoke tools securely while maintaining audit trails of tool usage.
- A2A Gateway: Enables secure agent-to-agent communication using the Agent-to-Agent protocol. Supports interoperability between agents from different frameworks, ensuring seamless coordination in distributed AI systems.
- Inference Routing: Intelligently routes requests to self-hosted models using Kubernetes Inference Gateway extensions. Compatible with inference servers like vLLM, TGI, and Triton, ensuring optimal performance for AI workloads.
- Supports regex-based filtering, OpenAI moderation, AWS Bedrock Guardrails, Google Model Armor, and custom webhooks. Ensures compliance with organizational policies and regulatory requirements by blocking or redacting sensitive content.
- Observability is enabled through OpenTelemetry, offering metrics, logs, and traces for monitoring traffic, latency, and costs.
- Spend Control: Enables organizations to attribute, price, and enforce spending limits on AI workloads. Supports per-model input and output pricing, ensuring that costs are tracked and controlled at the request level. Hard caps can be set per key or team, preventing budget overruns. The admin UI provides detailed spend reports, grouped by model, provider, and user.
- Deployment Flexibility: Agentgateway can be deployed in standalone mode or integrated into Kubernetes environments using Helm charts and the Gateway API. Supports both north-south and east-west traffic, with zero-config TLS, OIDC, and mTLS rotation. Compatible with Envoy for users requiring additional proxy capabilities.
- Admin UI & CLI: Includes a user-friendly admin UI for inspecting traffic, routing decisions, and policy enforcement. Features such as live request tracing and interactive terminal views enhance debugging and troubleshooting capabilities.
Technical Implementation & Releases
Agentgateway is implemented in Rust, ensuring high performance and reliability. The project follows a modular architecture, allowing it to handle diverse traffic types without requiring separate gateways. Recent releases have introduced significant enhancements, including improved LLM token counting, JWT authentication requirements, and support for new protocols and providers.
The v1.5.0 release introduced several breaking changes, such as normalized LLM token counting and stricter JWT issuer and audience claim requirements. These changes ensure consistency in token-based rate limiting and cost calculation across different LLM providers. The release also expanded support for Gemini inbound APIs, Anthropic Messages to OpenAI Responses conversion, and deeper integration with the Agent Substrate.
Key improvements include egress credential injection, support for Cohere Embed v4 responses, and enhancements to the admin UI and OpenTelemetry integration.
The project provides detailed documentation for both standalone and Kubernetes deployments, ensuring that users can configure and manage the proxy according to their specific requirements. The agctl CLI tool complements these deployment options by offering a command-line interface for interacting with the proxy’s configuration and monitoring its state.
The project’s roadmap includes ongoing development to address community feedback and emerging use cases. Contributions from developers, platform engineers, and AI enthusiasts are encouraged, with community meetings and GitHub issues serving as primary channels for collaboration. Agentgateway’s integration with the Linux Foundation ensures that it remains aligned with open-source best practices, fostering long-term adoption and innovation in AI infrastructure.