AI-Native Proxy Server and Data Plane for Agentic Applications
Plano is an AI-native proxy server and data plane designed to streamline the development and deployment of agentic applications. Originally developed as Arch Gateway, Plano centralizes critical infrastructure concerns such as agent routing, orchestration, smart LLM routing, prompt guardrails, and observability. By abstracting these components into a unified, out-of-process data plane, Plano eliminates the need for developers to repeatedly implement bespoke solutions for routing logic, safety moderation, and evaluation across different frameworks. This approach allows teams to focus on core product logic while ensuring scalability, reliability, and safety in production environments.
Built on the Envoy foundation, Plano leverages industry-proven infrastructure designed for modern workloads. Its architecture is purpose-built to handle the complexities of agentic systems, including low-latency orchestration between agents, model agility for routing decisions, and rich observability through agentic signals and OpenTelemetry (OTEL) traces. Plano’s design ensures that developers can integrate it seamlessly with any language or AI framework, accelerating the path to production without compromising flexibility or performance.
Plano’s core contributors bring experience from building critical infrastructure at scale, ensuring that the tool is optimized for real-world deployment challenges. The project is open-source under the Apache 2.0 license, fostering collaboration and customization within the AI community.
How It Works
Plano operates as a middleware layer that decouples developers from the operational overhead of managing agentic workflows. It achieves this through a modular architecture that handles orchestration, model management, and observability as configurable building blocks. Developers define their agents, models, and policies in a YAML configuration file, allowing Plano to dynamically route requests, enforce guardrails, and capture observability data without requiring additional instrumentation.
Agent Orchestration
Plano simplifies multi-agent workflows by allowing developers to declare agent descriptions, capabilities, and routing rules in a configuration file. For example, a travel assistant application can define separate agents for weather forecasts and flight searches, each exposed as an HTTP service. Plano’s orchestrator then determines which agent should handle a given request based on natural language descriptions and intent analysis. This eliminates the need to manually implement intent classifiers or routing logic, reducing complexity and improving maintainability.
Agents in Plano are implemented as HTTP servers that adhere to the OpenAI-compatible chat completions endpoint. Developers write their agent logic in their preferred language, while Plano handles the orchestration, model routing, and response streaming. For instance, a weather agent might fetch real-time data and stream responses back through Plano, which ensures consistent formatting and observability.
Smart LLM Routing
Plano’s model routing capabilities enable developers to define routing rules based on model names, aliases, or dynamic preferences. This flexibility allows applications to switch between different LLM providers seamlessly, such as OpenAI’s GPT-4o or Anthropic’s Claude-3-5-Sonnet, without modifying application code. If a specified model is unavailable, Plano can automatically fall back to a default or randomly select an alternative from configured providers.
Developers can also customize routing logic by replacing the default orchestrator with other models or fine-tuned variants.
Prompt Guardrails and Moderation
Plano integrates guardrail filters to enforce safety and moderation policies across all agentic interactions. These filters can detect and block jailbreak attempts, enforce content policies, and manage memory consistency.
Observability and Tracing
Observability is a core feature of Plano, providing end-to-end tracing and metrics for every request through OpenTelemetry. Plano automatically captures agentic signals, traces, and logs without requiring manual instrumentation. This data is exposed via OTLP/gRPC, enabling developers to analyze performance, debug issues, and optimize workflows. For instance, Plano can track latency spikes, model selection, and upstream response times, helping teams identify bottlenecks and improve efficiency.
Plano’s CLI tools, such as obs and trace, provide visibility into LLM traffic. The obs tool aggregates metrics like latency percentiles and throughput, while trace offers detailed insights into individual requests, including model selection and hop-by-hop timing.
Configuration-Driven Deployment
Plano’s configuration file serves as a single source of truth for defining agents, models, listeners, and policies. Developers specify agent URLs, natural language descriptions, model providers, and routing rules in YAML, allowing Plano to dynamically manage workflows. For example, a travel assistant configuration might include agents for weather and flights, each with descriptions of their capabilities. Plano then routes user queries to the appropriate agent based on intent, without requiring code changes.
Adding new agents or models involves updating the configuration file and restarting Plano, rather than redeploying application code. The configuration also supports environment variables for sensitive data, such as API keys, ensuring security and flexibility.
Use Cases
Plano is designed to address the operational challenges of building and scaling agentic applications. Its modular architecture makes it suitable for a variety of use cases, particularly in environments where reliability, safety, and observability are critical.
Multi-Agent Workflows
Plano excels in scenarios requiring coordination between multiple agents, such as customer support systems, travel assistants, or automated research tools. For example, a travel assistant might integrate weather, flight, and hotel agents to provide comprehensive trip planning. Plano’s orchestrator dynamically routes user queries to the appropriate agent, ensuring seamless workflows and reducing the complexity of managing inter-agent communication.
In the provided travel agent demo, Plano routes a user query about weather and flights to the respective agents without requiring custom routing logic. The agents focus on their specific tasks—fetching weather data or searching flights—while Plano handles the orchestration, response aggregation, and observability. This separation of concerns allows developers to iterate on agent logic independently while maintaining a unified user experience.
Model Agnosticism and Provider Switching
Plano enables applications to abstract away the complexities of interacting with multiple LLM providers. Developers can define routing rules in the configuration to direct requests to specific models based on cost, performance, or availability.
Plano’s model proxy standardizes access to LLMs, providing a consistent interface for all providers. This eliminates the need to handle provider-specific quirks, such as API differences or rate limits, in application code. Developers can switch providers or models by updating the configuration, enabling agility in response to pricing changes or model improvements.
Safety and Compliance
These filters can enforce policies such as content moderation, data privacy, or ethical guidelines. By centralizing these policies in filter chains, Plano ensures consistent enforcement across all agents and LLMs.
Observability and Continuous Improvement
By capturing traces, metrics, and agentic signals, developers can analyze system behavior, identify performance bottlenecks, and refine routing strategies.
The integration with OpenTelemetry ensures compatibility with existing observability tools, allowing teams to leverage their current infrastructure for monitoring and alerting. This reduces the overhead of implementing custom logging or tracing solutions, enabling faster iteration and deployment.
Key Features and Capabilities
Plano’s architecture is designed to address the operational pain points of agentic applications, offering a range of features that enhance efficiency, safety, and scalability.
Low-Latency Orchestration
Plano provides low-latency orchestration between agents, enabling real-time coordination in multi-agent workflows. The orchestrator dynamically routes requests based on agent descriptions and intent analysis, ensuring that users receive timely and relevant responses.
Model Agility
Developers can configure Plano to route requests by model name, alias, or dynamic preferences. This flexibility allows applications to adapt to changing requirements, such as switching to a more cost-effective model or leveraging new capabilities as they become available.
Agentic Signals and Observability
Plano captures rich agentic signals, including traces, metrics, and logs, without requiring manual instrumentation. These signals are automatically exported via OpenTelemetry, providing end-to-end visibility into agentic interactions.
Guardrail Filters for Safety and Moderation
These filters can detect and block jailbreak attempts, enforce content policies, and manage memory consistency. By centralizing guardrails in the configuration, Plano ensures that policies are applied uniformly, reducing the risk of compliance violations or security breaches.
Unified LLM APIs
Plano standardizes access to multiple LLM providers through a unified API interface. This abstraction eliminates the need to handle provider-specific quirks in application code, simplifying integration and reducing maintenance overhead. Developers can configure Plano to manage API keys, rate limits, and state management, ensuring consistent and reliable access to LLMs.
Zero-Code Agentic Signals
Plano automatically captures agentic signals, such as traces and metrics, without requiring developers to write custom instrumentation code. This feature accelerates development cycles by reducing the boilerplate associated with observability. The captured signals are compatible with OpenTelemetry, enabling seamless integration with existing monitoring tools.
Efficient Scaling
Plano’s architecture is optimized for scalability, allowing developers to add new agents or models with minimal effort. This modular approach reduces deployment complexity and enables teams to scale their applications more quickly.
Purpose-Built Lightweight LLMs
This approach ensures that Plano remains efficient and scalable even as the number of agents or models grows.
Cross-Platform Compatibility
Plano supports a range of platforms, including Linux (x86_64 and aarch64), macOS (Apple Silicon), and Docker. Developers can run Plano natively or within a containerized environment, depending on their infrastructure requirements. Pre-compiled binaries are automatically downloaded on first run, simplifying deployment and reducing setup time.
Developer Experience
Plano is designed to provide a delightful developer experience, with a simple configuration file that defines agents, models, and policies. The CLI tools, such as planoai up and planoai down, streamline management and interaction with the data plane. Additionally, Plano’s documentation and quickstart guides offer comprehensive resources for onboarding and troubleshooting.
Envoy Foundation
Plano is built on Envoy, a high-performance proxy and communication bus designed for modern distributed systems. Envoy’s architecture provides the scalability, reliability, and extensibility required for agentic applications. By leveraging Envoy’s WASM plugins and dynamic configuration, Plano can adapt to changing workloads and integrate with a wide range of services.
Session Cache and Latency Improvements
Recent optimizations, such as reducing the garbage collection (GC) slack in session cache time-to-live (TTL), have improved Plano’s performance. For example, release 0.4.36 addressed a +60-second delay in session cache TTL, enhancing responsiveness in high-throughput environments.
OpenTelemetry Integration
Plano’s integration with OpenTelemetry ensures compatibility with industry-standard observability tools. This reduces the complexity of implementing custom observability solutions and accelerates debugging and optimization.
CLI and Configuration Management
Plano’s CLI simplifies management tasks, such as starting, stopping, and configuring the data plane. The planoai up command, for example, automatically downloads dependencies like Envoy and WASM plugins, reducing setup time.
Security and Compliance
Plano addresses security concerns by centralizing guardrails and access controls. Additionally, Plano supports environment variables for sensitive data, such as API keys, ensuring secure handling of credentials.
Community and Contributions
Plano is an open-source project under the Apache 2.0 license, welcoming contributions from the AI community. Developers can participate in roadmap discussions, report bugs, or submit feature requests through the GitHub repository.