Introduction
NeMo Guardrails is an open-source toolkit developed by NVIDIA for integrating programmable guardrails into large language model (LLM) applications. Released in 2023 under the Apache 2.0 license, the toolkit is designed to enhance the safety, security, and controllability of conversational AI systems. It operates as a runtime defense mechanism, allowing developers to enforce policies, detect vulnerabilities, and ensure compliance with predefined rules. The toolkit is implemented in Python and supports a wide range of LLMs, including OpenAI’s GPT-3.5 and GPT-4, LLaMA-2, Falcon, Vicuna, and Mosaic.
The core of NeMo Guardrails lies in its ability to define and enforce guardrails at multiple stages of an LLM interaction: input, dialogue, retrieval, execution, and output. These guardrails are configured using the Colang language, a domain-specific language designed for modeling flexible yet controlled dialogue flows. Colang allows developers to define rules for user inputs, conversational paths, retrieved content, tool executions, and model outputs, ensuring that interactions adhere to organizational policies and safety standards.
The toolkit integrates with various safety and security models, including jailbreak detection systems, fact-checking mechanisms, and content moderation tools. It also supports third-party APIs and community models for enhanced functionality. NeMo Guardrails is structured to be modular, enabling developers to combine built-in guardrails with custom Python actions, external APIs, and proprietary models. This flexibility ensures that guardrails can be tailored to specific use cases, from customer support chatbots to domain-specific assistants and retrieval-augmented generation (RAG) systems.
How It Works
NeMo Guardrails operates by inserting a programmable layer between the application code and the LLM, intercepting and modifying interactions at critical stages. This layer is defined through a configuration system that includes YAML files for runtime settings, Colang flows for dialogue logic, and custom Python actions for application-specific checks. The toolkit provides two primary integration methods: a Python SDK for direct application integration and a guardrails server for HTTP/REST-based deployments compatible with OpenAI-style chat completions.
The guardrails configuration system is organized into five main types of rails, each targeting a specific stage of the LLM interaction pipeline:
- Input rails: Applied to user inputs to reject or alter messages based on predefined criteria, such as detecting jailbreak attempts or masking sensitive data.
- Dialog rails: Influence how the LLM is prompted by defining conversational flows, enforcing predefined responses, or guiding the interaction toward specific outcomes.
- Retrieval rails: Used in RAG scenarios to filter or modify retrieved content before it is passed to the LLM, ensuring factual accuracy and relevance.
- Execution rails: Validate and sanitize inputs and outputs of custom actions or tools called by the LLM, enhancing security in agentic workflows.
- Output rails: Applied to the LLM’s generated responses to block, alter, or validate content before it is returned to the user, ensuring compliance with safety and policy requirements.
The configuration process involves defining a guardrails configuration folder containing a config.yml file for general settings, a rails.co file for Colang definitions, and optional Python scripts for custom logic. For example, a config.yml might specify the LLM model, active rails, and sensitivity detection rules, while a rails.co file could define dialogue flows for greeting users or handling insults. The toolkit supports both Colang 1.0 and 2.0, with 1.0 being the default.
This ensures consistency across different guardrail types and integration points, such as LLMRails and IORails. The toolkit’s architecture is designed to be portable, allowing configurations developed locally with the Python SDK to be deployed seamlessly in production using the NeMo Guardrails microservice, which provides a production-ready container image.
The toolkit’s async-first design ensures efficient handling of concurrent requests, with both synchronous and asynchronous APIs available for integration. For instance, the LLMRails.generate and LLMRails.generateasync methods allow developers to invoke guardrailed LLM interactions with minimal code changes. The input and output formats for these methods align with the OpenAI Chat Completions API, facilitating adoption in existing workflows.
Use Cases
NeMo Guardrails is applicable across a broad range of LLM-based applications, particularly those requiring controlled, safe, and policy-compliant interactions. Key use cases include:
- Question Answering over Documents (RAG): Enforces fact-checking and output moderation to ensure responses are accurate, relevant, and free from hallucinations or biased content. Retrieval rails can filter or modify retrieved chunks to exclude sensitive or irrelevant information before it influences the LLM’s response.
- Domain-Specific Assistants (Chatbots): Ensures assistants adhere to predefined conversational flows and topic boundaries. For example, a customer support bot can be configured to avoid political discussions while guiding users through authentication and support procedures. Dialog rails can enforce greeting protocols, handle insults with predefined responses, and steer conversations toward resolution.
- LLM Endpoints: Adds guardrails to custom LLM endpoints to enhance customer interaction safety. Output rails can block harmful or off-topic responses, while input rails can detect and mitigate jailbreak attempts.
- LangChain Chains: Integrates with LangChain frameworks to add guardrails around existing chains. By setting the NEMOGUARDRAILSLLMFRAMEWORK=langchain environment variable, developers can wrap LangChain chains with guardrails, ensuring that tool calls, agent workflows, and multi-agent interactions are validated and secured. This integration is critical for agentic systems where tool validation and workflow protection are essential.
Beyond these primary use cases, NeMo Guardrails addresses specialized security and compliance requirements, such as content safety, jailbreak protection, topic control, and personally identifiable information (PII) detection. For instance, content safety guardrails can integrate with NVIDIA’s Llama 3.1 NemoGuard 8B model for robust moderation or leverage third-party APIs like ActiveFence and Cisco AI Defense. Jailbreak protection layers combine self-check detection, heuristic pattern matching, and advanced threat detection models like NemoGuard Jailbreak Detection NIM.
In agentic security scenarios, NeMo Guardrails provides tool call validation, workflow protection, and secure tool integration. It supports integrations with LangGraph for multi-agent safety, ensuring that agent actions are monitored, validated, and sanitized. Execution rails can validate tool inputs and outputs, while action monitoring enables detailed logging and tracing of agent behavior. This is particularly valuable in environments where LLMs interact with external systems, requiring strict isolation of authentication information and validation of all inputs.
Key Features & Capabilities
NeMo Guardrails offers a comprehensive set of features designed to enhance the safety, security, and controllability of LLM applications. These features are built around its modular architecture, which combines built-in guardrails, custom actions, and third-party integrations.
Guardrail Types and Configuration
The toolkit supports five distinct guardrail types, each addressing a specific stage of the LLM interaction pipeline. Input rails, for example, can reject or alter user inputs based on sensitivity detection or jailbreak patterns. Dialog rails define conversational flows using Colang, allowing developers to enforce greeting protocols, handle specific user expressions, or guide interactions toward predefined outcomes. Retrieval rails are essential in RAG scenarios, where they filter or modify retrieved content to ensure relevance and accuracy. Execution rails validate tool calls and outputs, while output rails enforce safety and policy compliance on the LLM’s responses.
Guardrail configurations are defined in YAML files, which specify models, active rails, and custom settings. Colang flows, written in .co files, define the logic for dialogue management and event-driven behavior. For instance, a greeting.co file might define flows for greeting users and offering assistance, while an insult.co file could specify responses to offensive language.
Built-in Guardrails and Integrations
NeMo Guardrails includes a suite of built-in guardrails for common safety and security challenges.
- LLM Self-Checking: Uses the LLM itself to validate inputs and outputs for harmful content, factual accuracy, and hallucination risks.
- NVIDIA Safety Models: Integrates with models like Llama 3.1 NemoGuard 8B for content safety and topic control, providing robust moderation capabilities.
- Community Models: Supports integrations with LlamaGuard, Fiddler Guardrails, and other community-driven safety solutions.
- Third-Party APIs: Connects to services like ActiveFence, Cisco AI Defense, and Prompt Security for advanced moderation and threat detection.
- Jailbreak Detection: Combines self-check mechanisms, heuristic pattern matching, and advanced models like NemoGuard Jailbreak Detection NIM to prevent adversarial manipulations.
These built-in guardrails can be combined with custom Python actions, LangChain tools, or third-party APIs to create tailored safety and compliance solutions.
Runtime Interfaces and Deployment
NeMo Guardrails provides two primary runtime interfaces for integration: the Python SDK and the guardrails server. The Python SDK allows developers to add guardrails directly into their applications with minimal code changes. For example, loading a guardrails configuration and creating an LLMRails instance involves two steps: loading the configuration and invoking the generate or generateasync methods. The input and output formats for these methods align with the OpenAI Chat Completions API, ensuring compatibility with existing workflows.
The guardrails server offers an HTTP/REST-based deployment option, exposing endpoints like /v1/chat/completions for guardrailed chat completions. This server is compatible with OpenAI-style APIs, making it easy to integrate into existing infrastructure. The server also supports health endpoints, streaming requests, and tool calling, enhancing its suitability for production environments. Configurations developed with the Python SDK are portable to the guardrails server, enabling seamless transitions from development to deployment.
Performance and Scalability
The toolkit’s async-first architecture ensures efficient handling of concurrent requests, while features like shared outbound HTTP clients and connection pooling optimize resource usage. Recent releases, such as version 0.24.0, have expanded the capabilities of IORails, allowing it to execute 59 of the library’s 67 action-backed input and output surfaces. This includes integrations with community models and rails that transform content, all while sharing underlying implementations with LLMRails.
The toolkit also introduces improvements like RailOutcome, which standardizes the contract for rail actions, and rail manifests, which declare configuration, actions, and execution surfaces. These foundations enable both LLMRails and IORails to discover and use built-in rails without maintaining separate support lists. Additionally, the toolkit supports observability features like OpenTelemetry for content capture and request tracing, providing insights into guardrail performance and behavior.
Versioning and Compatibility
NeMo Guardrails follows a structured release cycle, with the latest stable version being 0.24.1 as of September 2026. This release includes bug fixes and improvements, such as corrections for Locust concurrency measurements, parsing Nemotron 3.5 responses, and handling IORails admission shedding.
The toolkit is compatible with Python 3.10 through 3.13 and supports a wide range of LLMs, including OpenAI’s models, LLaMA-2, Falcon, Vicuna, and Mosaic. Developers can install NeMo Guardrails via pip, with detailed installation guides available in the documentation. The toolkit’s modular design ensures that configurations and guardrails can be reused across different deployment environments, from local development to production microservices.
Conclusion
NeMo Guardrails stands out as a versatile and powerful toolkit for developers seeking to enhance the safety, security, and controllability of LLM-based applications. Its modular architecture, combined with support for custom actions, third-party integrations, and built-in guardrails, provides a comprehensive solution for addressing a wide range of challenges in conversational AI. From enforcing content safety and preventing jailbreaks to managing PII detection and agentic workflows, NeMo Guardrails offers the flexibility and robustness required for production-grade deployments.
The toolkit’s integration with popular frameworks like LangChain and LangGraph further extends its applicability, making it a valuable asset for teams working with multi-agent systems and complex tool-based interactions. Its async-first design and support for both Python SDK and server-based deployments ensure scalability and performance, while features like RailOutcome and rail manifests streamline the development and maintenance of guardrail configurations. As LLM applications continue to evolve, NeMo Guardrails provides a foundation for building trustworthy, secure, and compliant AI systems.