Span-Level Hallucination Detection for RAG and Tool-Grounded Outputs
LettuceDetect is an open-source span-level hallucination detector designed to identify unsupported, contradictory, or fabricated segments in AI-generated outputs. It operates by comparing generated answers against provided context, marking specific character spans as hallucinated rather than providing only an overall verdict. The tool is optimized for real-time grounding checks during inference, making it suitable for integration into retrieval-augmented generation (RAG) systems and tool-grounded workflows.
The project builds on encoder-based models inspired by the Luna paper, which employs a token-level approach to detect hallucinations. LettuceDetect extends this methodology to support a broader range of use cases, including multilingual question answering, code-agent responses, and developer-tool outputs. Its architecture includes both fast local encoder models and generative detectors capable of categorizing detected spans.
LettuceDetect was introduced by KR Labs in 2025 and has since evolved to address hallucination detection across diverse AI workflows. The tool is released under the MIT License, ensuring permissive use in both research and production environments. It is implemented in Python and can be deployed as a library or accessed via an API, making it versatile for developers and security researchers.
How It Works
LettuceDetect operates by analyzing AI-generated answers at the token or span level, comparing them against provided context to identify discrepancies. The tool supports two primary output formats: span format, which returns hallucinated text segments along with their start and end positions in the answer, and token format, which labels individual tokens as supported or hallucinated with associated confidence scores.
The span format is particularly useful for pinpointing exact locations of hallucinations, while the token format provides granularity for further analysis. Users can adjust confidence thresholds to filter predictions, ensuring only high-confidence hallucinations are flagged.
For inference, LettuceDetect provides a Python API that can be integrated into existing systems with minimal setup. The tool also includes a web API and a Streamlit demo for interactive exploration. Models are hosted on Hugging Face, allowing users to load them with a single line of code via the transformers library.
Use Cases
LettuceDetect is primarily used for testing and red-teaming AI systems, particularly in environments where grounding and factual accuracy are critical. Its span-level precision makes it ideal for identifying hallucinations in RAG outputs, where generated answers must align closely with retrieved context. The tool is also effective for evaluating code-agent responses, tool-grounded outputs, and structured documents, where even minor inaccuracies can have significant consequences.
One of the key applications of LettuceDetect is in the development and validation of coding agents. These agents generate code based on repository sources and developer-tool outputs, and LettuceDetect can detect invented APIs, incorrect identifiers, or unsupported behaviors. For instance, the tool can flag a coding agent’s response that introduces a non-existent function or returns an incorrect value, ensuring that only grounded and accurate outputs are accepted.
Beyond code and RAG, LettuceDetect supports multilingual question-answering systems, where hallucinations can lead to misleading or incorrect information. Additionally, LettuceDetect can be used in conjunction with other verification tools, such as triplet-based fact-checkers, to provide layered validation of AI outputs.
The tool’s efficiency and low latency make it suitable for production environments. For example, TinyLettuce, a variant of LettuceDetect, uses Ettin encoders with as few as 17 million parameters to achieve real-time hallucination detection on CPU hardware. This reduces the computational overhead associated with larger models while maintaining competitive accuracy. TinyLettuce is particularly valuable in resource-constrained deployments where speed and efficiency are priorities.
Key Features and Capabilities
- Token-Level Precision: LettuceDetect identifies exact hallucinated spans within an answer, rather than providing a binary verdict. This granularity allows developers to pinpoint specific segments of text that lack grounding in the provided context, facilitating targeted corrections or revisions.
- Support for Diverse Output Types: The tool extends beyond traditional RAG prose to include code-agent answers, tool outputs, and structured documents.
- Multilingual Support: LettuceDetect includes models trained on multilingual datasets, enabling hallucination detection in seven languages.
- This performance ensures that grounding checks can be performed in real time without significant delays.
- This capability is essential for handling the extensive contexts often encountered in RAG systems, where retrieved documents may contain thousands of tokens.
- Typed Spans in v2 Models: The latest versions of LettuceDetect include generative models that categorize detected spans into hallucination types and subcategories.
- Integration with Developer Tools: LettuceDetect can be integrated into coding workflows, such as Claude Code, where it acts as a Stop hook to validate agent answers against grounding context.
- Benchmarking and Evaluation: The tool includes scoped benchmark results and model cards that document performance metrics, limitations, and evaluation methodologies.
- Synthetic Data Generation: LettuceDetect provides a pipeline for generating synthetic training data, enabling users to create domain-specific models tailored to their needs. This feature reduces the reliance on manual annotation and accelerates the development of specialized detectors.
- TinyLettuce Variants: For deployments with limited resources, TinyLettuce offers lightweight Ettin encoder models with 17–68 million parameters. These models achieve competitive accuracy while running in real time on CPU hardware, making them ideal for edge devices or environments with constrained computational resources.
- API and Library Support: LettuceDetect can be deployed as a Python library or accessed via a web API. The tool also provides a Streamlit demo for interactive exploration and testing.
The tool’s architecture includes a hierarchy of detectors, allowing users to select between transformer-based and LLM-based methods depending on their requirements. For example, the LLMDetector can be used for zero-shot or few-shot hallucination detection, while transformer-based detectors offer faster inference with lower computational overhead.
Model Variants and Performance
The lettucedect-v2-qwen-2b is a generative model that emits typed spans in a single pass, providing detailed categorization of hallucinations. The lettucedect-v2-mmbert-base is a fast encoder model optimized for local inference, while the lettucedect-v2-taxonomy-head extends the taxonomy cascade for typed span detection.
For multilingual applications, LettuceDetect provides EuroBERT-based models in two sizes: a base variant with 210 million parameters for faster inference and a large variant with 610 million parameters for higher accuracy.
TinyLettuce introduces even smaller models, with variants ranging from 17 million to 68 million parameters. These models are trained on synthetic data generated using LettuceDetect’s HallucinationGenerator class, which creates controllable error types for targeted training. TinyLettuce models achieve an F1 score of 90.87% on synthetic test data, outperforming larger models like GPT-5-mini and Qwen3-235B while running in real time on CPU hardware.
The performance of LettuceDetect has been benchmarked against other hallucination detection methods, including GPT-4, Luna, and fine-tuned Llama-2-13B models. In evaluations on the RAGTruth test set, LettuceDetect surpasses these baselines, demonstrating its effectiveness in identifying unsupported spans in AI-generated outputs. The tool’s synthetic data generation pipeline further enhances its adaptability, allowing users to train models on domain-specific datasets.
Integration and Deployment
LettuceDetect can be installed via pip and integrated into existing systems with minimal setup. The tool provides a Python API that supports both span and token output formats, allowing users to customize predictions based on confidence thresholds. For example, the following code snippet demonstrates how to use the API to detect hallucinations in an answer:
predictions = detector.predict(contexts=contexts, question="", answer=answer, output_format="spans")
This code will return a list of hallucinated spans, including their start and end positions, confidence scores, and the hallucinated text. Users can adjust the minconfidence parameter to filter predictions based on their desired level of certainty.
For deployment in production environments, LettuceDetect includes a web API that can be started using the startapi.py script. The API supports both development and production modes, with options for customizing the model and detection method. The following command demonstrates how to start the API in production mode:
python scripts/startapi.py prod --model KRLabsOrg/lettucedect-base-modernbert-en-v1
The API can then be accessed via a Python client library, which provides methods for detecting spans and tokens in AI-generated outputs. For example, the LettuceClient class allows users to send requests to the API and retrieve predictions in real time.
LettuceDetect also supports integration with coding agents and developer tools. For instance, the tool can be used as a Stop hook in Claude Code to validate agent answers against grounding context.
Benchmarking and Evaluation
LettuceDetect includes comprehensive benchmarking and evaluation methodologies to assess its performance across different use cases. These benchmarks provide insights into the tool’s accuracy, precision, and recall in identifying unsupported spans in AI-generated outputs.
The evaluation process includes example-level, token-level, and span-level assessments, allowing users to compare performance across different granularities. For instance, the tool’s F1 score is reported at the example level, reflecting whether an answer contains any hallucinations versus none.
LettuceDetect’s performance has been compared against other hallucination detection methods, including LLM judges like GPT-4.1-mini and fine-tuned models such as Llama-2-13B. In these comparisons, LettuceDetect consistently outperforms baseline models, demonstrating its superiority in identifying unsupported spans while maintaining low computational overhead.
The tool’s synthetic data generation pipeline further enhances its adaptability, allowing users to create domain-specific training datasets. This capability is particularly valuable for applications where manual annotation is impractical or costly. By generating synthetic data with controllable error types, users can train models tailored to their specific use cases, improving accuracy and relevance.
Community and Collaboration
LettuceDetect is an open-source project maintained by KR Labs, with contributions from the broader AI security community. The project includes a public roadmap that outlines upcoming features and research directions, inviting collaboration from developers, researchers, and security professionals. Users can engage with the community via GitHub Discussions for use-case exploration and research questions, or join the Discord server for informal conversations.
New contributors are encouraged to explore good first issues and help-wanted tasks, while active work is clearly labeled to avoid duplication of effort.
LettuceDetect’s MIT License ensures permissive use, allowing developers to integrate the tool into proprietary systems without legal restrictions. The project’s documentation, including model cards and benchmark results, is openly available, providing transparency into its capabilities and limitations.