SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
SelfCheckGPT is a Python-based tool designed for testing and red teaming generative large language models (LLMs) to detect hallucinations without requiring direct access to the model’s internal parameters. The project was introduced in November 2023 and is licensed under the MIT License, allowing for broad use and modification. It was developed by the POTSAWEE research group and is intended for researchers, developers, and security professionals working with LLMs.
It operates as a black-box solution, meaning it does not rely on the internal workings of the LLM being tested.
The paper explores multiple variants of the self-check approach, including BERTScore, Question-Answering (MQAG), n-gram overlap, Natural Language Inference (NLI), and LLM-Prompting.
The tool is actively maintained, with updates released in early 2024 adding support for LLM-Prompting via OpenAI’s API and open-source models like Mistral and Llama. These updates expand its applicability to a wider range of use cases, including real-time evaluation of commercial and open-source LLMs.
How It Works
SelfCheckGPT operates by generating multiple versions of a passage using the same LLM under evaluation. These versions are then compared to identify inconsistencies or factual inaccuracies. The core assumption is that hallucinations will manifest as discrepancies between the original passage and its sampled alternatives.
BERTScore
It uses a pre-trained BERT model to compute embeddings and calculates a score between 0.0 and 1.0, where higher values indicate a higher likelihood of hallucination.
Example usage involves splitting the passage into sentences, then comparing each sentence against the sampled passages. The output is a list of scores corresponding to each sentence, reflecting its consistency with the sampled data.
SelfCheck-MQAG (Question-Answering)
- Counting: Uses an answerability threshold to determine factual consistency by counting how often questions yield consistent answers across passages.
Scores range from 0.0 to 1.0, with higher values indicating non-factual content.
SelfCheck-Ngram
It calculates negative log probabilities for each sentence and document, where higher values suggest a higher likelihood of hallucination.
The output includes both sentence-level and document-level scores, providing granular insights into where hallucinations may occur within a passage.
SelfCheck-NLI (Natural Language Inference)
SelfCheck-NLI uses a fine-tuned DeBERTa-v3-large model to classify sentences as either entailing, contradicting, or being neutral with respect to the sampled passages. The probability of contradiction is normalized and used as the hallucination score. This approach is computationally efficient and has shown strong performance in detecting factual inaccuracies.
SelfCheck-LLM Prompting
The most recent addition to SelfCheckGPT, LLM-Prompting, involves querying another LLM (such as Mistral, Llama, or OpenAI’s GPT models) to assess whether a given sentence is supported by the sampled context. The default prompt template is structured as follows:
Is the sentence supported by the context above?
The LLM’s response is converted into a binary score (Yes = 0.0, No = 1.0, with ambiguous answers scored as 0.5). This method is highly flexible, as it can be adapted to any Hugging Face-hosted model and supports both API-based and local inference.
Use Cases
- Red Teaming: Security professionals and AI ethics researchers use the tool to identify vulnerabilities in LLMs, such as generating factually incorrect or misleading information.
- This allows for objective assessments of which models are more reliable in generating grounded responses, aiding in the selection of models for specific tasks.
- The tool’s open-source nature makes it accessible for experimental purposes.
- By cross-referencing outputs with multiple sampled passages, the tool helps mitigate the spread of misinformation in automated journalism or social media moderation.
The tool’s compatibility with both open-source and proprietary LLMs (via API) expands its utility across diverse environments, from local development setups to cloud-based enterprise solutions.
Key Features & Capabilities
- Zero-Resource Black-Box Design: The tool does not require access to the internal parameters or training data of the LLM being evaluated. This makes it applicable to any generative model, regardless of its architecture or proprietary status.
- Sentence-Level Granularity: All methods provide scores at the sentence level, enabling precise identification of where hallucinations occur within a passage. This granularity is essential for debugging and refining LLM outputs.
- Support for Large-Scale Evaluation: The tool can process entire documents by aggregating sentence-level scores into document-level metrics.
- Integration with Existing Workflows: SelfCheckGPT is designed to integrate seamlessly with Python-based pipelines. It supports common libraries like torch, spacy, and transformers, and can be installed via pip install selfcheckgpt. Example workflows are provided in Jupyter Notebooks, facilitating adoption by developers and researchers.
- Dataset for Benchmarking: The tool includes the wikibiogpt3hallucination dataset, which consists of 238 annotated passages comparing GPT-3-generated text with corresponding Wikipedia entries. This dataset serves as a ground truth for evaluating the effectiveness of SelfCheckGPT’s methods and can be loaded directly via the Hugging Face datasets library or downloaded in JSON format.
- Reproducibility: The tool includes mechanisms for setting random seeds (e.g., torch.manual_seed) to ensure consistent results across multiple runs.
- API and Open-Source Model Support: The LLM-Prompting variant supports both API-based models (OpenAI, Groq) and open-source models hosted on Hugging Face.
Implementation and Technical Details
SelfCheckGPT is implemented in Python and leverages PyTorch for computational efficiency.
- Device Agnosticism: The tool automatically detects and utilizes GPU acceleration if available, optimizing performance for large-scale evaluations.
- Modular Architecture: Each detection method is encapsulated in a separate class (e.g., SelfCheckMQAG, SelfCheckBERTScore), allowing users to mix and match approaches based on their needs. The predict() method standardizes the interface across all variants.
- Error Handling and Ambiguity Resolution: The LLM-Prompting method includes mechanisms to handle ambiguous responses (e.g., non-binary outputs) by assigning a midpoint score (0.5), ensuring robustness in real-world scenarios.
- For example, Release 0.1.7 (March 2024) introduced OpenAI API support for LLM-Prompting, while Release 0.1.4 (July 2023) added SelfCheck-NLI and baseline rescaling for BERTScore.