How it Works
New API is a self-hosted AI gateway designed to aggregate multiple AI model providers behind a unified OpenAI-compatible API. It acts as an intermediary layer that standardizes access to diverse upstream services such as OpenAI, Anthropic, Google Gemini, Azure OpenAI, AWS Bedrock, Vertex AI, DeepSeek, Qwen, and other compatible providers. By consolidating these services, New API enables organizations to manage routing, access control, quotas, billing, and format conversion from a single interface.
Each channel represents a connection to a specific AI service, allowing users to define model mappings, priority settings, retries, and affinity rules. This setup ensures that requests are dynamically routed to the most appropriate upstream service based on predefined criteria. For example, a channel can be configured to prioritize a specific model from one provider while falling back to another if the primary service is unavailable or overloaded.
New API supports protocol conversion between different AI service APIs, including OpenAI Chat Completions, Anthropic Messages, and Gemini APIs. While not all features may be supported across all conversion paths, RelayKit ensures compatibility where possible, allowing seamless interaction between disparate AI services.
The gateway also includes a web console for configuration, monitoring, and management.
At its core, New API is built using Go for the backend and React 19 with TypeScript for the frontend. The architecture is modular, allowing for extensions and customizations. For instance, asynchronous task APIs can be extended with JavaScript plugins to handle image, video, and other long-running processes. These plugins enable features such as task status tracking, output retrieval, and integration with external services.
The system is designed to be deployed in various environments, from local development to production-grade setups. It supports multiple database backends, including SQLite, MySQL, and PostgreSQL, as well as optional Redis for caching and rate limiting. Deployment can be as simple as running a Docker container for local testing or as complex as a multi-node distributed setup for high availability and load balancing.
Users can manage API keys, restrict access to specific models or groups, and ensure that all interactions comply with upstream service terms and applicable regulations. The system also includes features for logging, audit trails, and cost accounting, which are essential for organizations operating AI services at scale.
New API is licensed under AGPL 3.0, a copyleft license that ensures the software remains open source while requiring derivative works to also be open source.
Use Cases
New API is primarily designed for organizations and teams that require centralized management of AI model access. One of its key use cases is enabling teams to share authorized access to multiple AI models without requiring individual clients to reconfigure their integrations every time a new provider or model is added. This is particularly useful in environments where AI services are frequently updated or where multiple providers need to be accessed simultaneously.
Another significant use case is the operation of private multi-model services. Organizations can deploy New API to create an internal AI hub that provides a unified interface for all their AI-related needs. This setup allows for centralized control over usage, costs, and access, while also enabling features such as model routing, quota management, and billing. For example, a research lab might use New API to aggregate access to various large language models, ensuring that all team members can interact with these models through a single, consistent API.
New API is also well-suited for developers and organizations that need to build AI-powered applications with support for multiple providers. By exposing a unified OpenAI-compatible API, New API simplifies the integration process, allowing developers to focus on building applications rather than managing the complexities of different AI service APIs. This is particularly valuable in scenarios where applications need to switch between providers or models dynamically based on factors such as cost, performance, or availability.
For organizations operating AI services as a public offering, New API provides the infrastructure needed to comply with regulatory requirements. Additionally, the system supports asynchronous task processing, which is useful for handling long-running operations such as image or video generation, where users may need to check the status of their requests or retrieve outputs at a later time.
In enterprise environments, New API can serve as a cost optimization tool. By centralizing access to multiple AI providers, organizations can compare pricing, usage patterns, and performance metrics across different services. This allows them to make informed decisions about which models to use for specific tasks, ultimately reducing costs while maintaining high-quality outputs. The system’s quota management and billing features further enhance this capability by providing granular control over usage and spending.
Finally, New API is useful for developers and organizations that want to extend the functionality of AI services through custom plugins. The experimental plugin system, introduced in recent releases, allows users to create JavaScript-based extensions for handling tasks such as image generation, video processing, and other asynchronous operations. These plugins can be integrated into the gateway to provide additional features or to interface with external systems, making New API a flexible platform for building custom AI workflows.
Model Access and Protocol Support
New API supports a wide range of AI model interfaces, including OpenAI Chat Completions, Anthropic Messages, and Google Gemini APIs. This compatibility ensures that existing applications can integrate with New API with minimal changes.
The system also supports streaming responses and WebSocket connections, which are essential for real-time interactions with AI models.
Routing and Load Management
Routing is a core feature of New API, enabling users to define how requests are distributed across multiple upstream providers. Users can configure model mappings, channel priorities, weights, retries, and affinity rules to ensure that requests are routed optimally. This flexibility allows organizations to balance performance, cost, and availability.
New API also supports channel affinity, which ensures that requests from a specific client or session are consistently routed to the same upstream service. Additionally, the system includes retry mechanisms that automatically reroute failed requests to alternative providers, improving reliability and reducing downtime.
Usage and Cost Management
One of New API’s standout capabilities is its comprehensive usage and cost management system.
New API also provides detailed usage logs and analytics, which help organizations monitor their AI consumption and optimize costs. These logs include information about request volume, response times, and billing details, allowing users to track spending and identify areas for improvement.
Access Control and Security
Security is a critical aspect of New API, and the system includes multiple layers of access control. Users can manage individual accounts, groups, and API keys with fine-grained permissions. For example, an administrator might create a group for a specific team and restrict access to only the models and features that team requires.
Users are responsible for obtaining valid API keys and ensuring that their usage complies with legal requirements, particularly when operating AI services for public consumption.
Asynchronous Tasks and Plugins
New API supports asynchronous task processing through JavaScript plugins, which extend the system’s functionality beyond standard API calls. These plugins can handle long-running operations such as image generation, video processing, and other computationally intensive tasks. For example, a plugin might be used to manage the lifecycle of an image generation task, including submission, status tracking, and output retrieval.
This feature is particularly useful for organizations that need to integrate image generation into their workflows while maintaining control over costs and usage.
Plugins can also be used to extend the system’s routing and conversion capabilities. For example, a plugin might be developed to handle custom model formats or to interface with proprietary AI services. The experimental plugin system, while still under development, provides a framework for building these extensions, allowing users to tailor New API to their specific needs.
Deployment and Scalability
New API is designed to be deployed in a variety of environments, from local development to large-scale production systems. The system supports multiple deployment methods, including Docker containers, Docker Compose, and multi-node distributed setups. For example, users can deploy New API locally using a single Docker container with SQLite for development and testing. For production environments, a Docker Compose setup with PostgreSQL and Redis is recommended to ensure scalability and reliability.
Multi-node deployments allow organizations to distribute the load across multiple servers, improving performance and availability. In such setups, nodes must share a common database and authentication secrets to maintain consistency. The system also supports reverse proxy configurations, which are essential for securing the web console and enabling HTTPS connections. Users can configure trusted proxies, session secrets, and cookie security settings to ensure that their deployment is secure and compliant with best practices.
Web Console and User Interface
The New API web console provides a comprehensive interface for managing the gateway, including configuration, monitoring, and analytics. Key features of the console include a model playground for testing interactions, a usage analytics dashboard, and a configuration editor for defining routing rules and access policies.
The console also includes tools for managing asynchronous tasks, such as viewing task statuses and retrieving outputs. Users can monitor the performance of plugins and upstream services, ensuring that their AI workflows are running smoothly.
Technical Implementation
New API is implemented using Go for the backend and React 19 with TypeScript for the frontend. The backend leverages the Gin web framework, while the frontend uses modern tools such as Rsbuild, TanStack, and Tailwind CSS for building a responsive and efficient user interface. The system is containerized using Docker, making it easy to deploy and scale across different environments.
The architecture of New API is modular, with distinct components for routing, access control, billing, and plugin management. This modularity allows users to extend the system’s functionality by adding custom plugins or integrating with external services.
Recent Developments and Future Directions
Recent releases of New API have introduced several new features and improvements, particularly in the areas of task plugins, routing, and cost management. Additionally, the plugin system has been enhanced to handle OpenAI Images and other asynchronous tasks more efficiently, with better support for upstream status codes and pricing configurations.
Updates to the routing system have improved the clarity of model mappings and channel configurations, making it easier for users to manage their AI services. The introduction of request policies has added an additional layer of control, allowing organizations to define rules for request checks, session management, and retry behavior. These policies are logged in usage logs, providing transparency and accountability.
The project’s open-source nature ensures that it will continue to evolve based on community feedback and contributions, making it a dynamic and adaptable platform for AI gateway management.