How it Works
Loki is an open-source fact-checking pipeline developed by Libr-AI that automates the verification of factual claims in text. The tool decomposes input text into individual claims, evaluates their verifiability, generates evidence retrieval queries, and searches for supporting evidence using web-based search engines. Once evidence is gathered, Loki employs a language model to assess the accuracy of each claim, producing a structured report detailing the verification process.
The pipeline operates through a modular architecture, where each component—claim extraction, evidence retrieval, and verification—is encapsulated in distinct classes. This design allows for flexibility in customization, such as switching between different language models (e.g., OpenAI or Anthropic), search engines (e.g., Serper or Google), or predefined prompts. The tool supports both programmatic use via a Python library and a web application interface, accommodating diverse deployment needs.
Configuration is managed through environment variables or a YAML file, where users specify API keys for search engines and language models. This approach ensures security and ease of use, particularly for teams automating factuality checks in generated content. Loki’s architecture abstracts underlying complexities, allowing users to focus on customizing specific components—such as model selection or prompt definitions—without extensive technical overhead.
The verification process begins with claim extraction, where the input text is parsed into discrete statements. Evidence retrieval queries are generated and executed using the selected search engine, fetching relevant web sources. Finally, the language model evaluates the retrieved evidence against the original claim, producing a report that includes the claim, evidence, and verification outcome.
Users can define custom prompts in YAML or JSON format, enabling tailored verification logic for specific use cases.
These updates reflect a commitment to maintaining a scalable and maintainable codebase, aligning with the tool’s purpose of supporting teams in red-teaming and testing AI-generated content for factual accuracy.
Use Cases
Loki is primarily designed for teams engaged in testing and red-teaming AI-generated content, particularly in identifying hallucinations and assessing groundedness. Its structured approach to fact-checking makes it valuable for journalists, researchers, and organizations seeking to automate the verification of information. By decomposing text into verifiable claims and retrieving evidence from the open web, Loki helps mitigate the spread of misinformation in generated outputs.
One key application of Loki is in the evaluation of large language models (LLMs) and other AI systems. Developers and security teams can use the tool to systematically verify the factual accuracy of AI responses, identifying instances where models produce incorrect or unsupported claims. This capability is critical in red-teaming exercises, where the goal is to uncover vulnerabilities in AI systems by simulating adversarial scenarios.
Researchers and academics benefit from Loki’s ability to process large volumes of text efficiently. The tool’s automation reduces the manual effort required for fact-checking, allowing researchers to focus on analysis rather than evidence retrieval. For example, a study analyzing AI-generated summaries could use Loki to cross-reference claims with reliable sources, ensuring the integrity of the research findings. Similarly, journalists can leverage Loki to fact-check articles or reports, enhancing the credibility of their work through automated verification.
Loki’s open-source nature and permissive MIT license make it accessible to a wide range of users, from individual developers to large organizations. The tool’s modular design further encourages community contributions, with users able to extend its functionality through custom prompts, models, or search engines.
For teams focused on red-teaming or automated fact-checking, Loki provides a scalable solution that balances accuracy with efficiency, addressing the growing need for trustworthy information in AI-driven environments.
Key Features & Capabilities
Loki’s core functionality revolves around its fact-checking pipeline, which consists of claim extraction, evidence retrieval, and verification. The tool begins by dissecting input text into individual claims, a process that involves parsing sentences and identifying statements that can be factually verified. Each claim is then assessed for its potential to be supported or refuted by external evidence, with low-priority claims excluded to optimize processing time.
Evidence retrieval is a critical component of Loki’s pipeline, where the tool generates queries based on the extracted claims and searches the open web for relevant sources. The retrieved evidence is then compiled into a structured format for further analysis.
Verification is performed using a language model, with Loki supporting both OpenAI and Anthropic APIs. The language model evaluates the retrieved evidence against the original claim, determining whether the claim is supported, refuted, or lacks sufficient evidence. The results are compiled into a detailed report, which includes the claim, evidence sources, and the verification outcome, providing transparency into the fact-checking process.
Loki’s modular architecture is a defining feature, with each component—such as claim extraction, evidence retrieval, and verification—implemented as a separate class. The tool also supports prompt definitions via YAML or JSON files, enabling users to define and switch between different verification strategies dynamically.
Configuration management is streamlined through environment variables or a YAML file, where users specify API keys for search engines and language models.
This feature extends the tool’s applicability beyond traditional text-based verification, enabling comprehensive analysis of multimedia content.
Loki’s release history highlights continuous improvements, including the introduction of a --model parameter for model switching, support for multiple search engines, and enhancements to the web application interface.
The project’s open-source nature and permissive MIT license encourage collaboration and customization. Users can contribute to the development of Loki by extending its functionality, reporting bugs, or proposing new features. This openness fosters a community-driven approach to fact-checking, where collective efforts enhance the tool’s accuracy and applicability across diverse use cases.
Loki’s integration with web search and language models ensures that evidence retrieval and verification are grounded in real-world data. The structured report generated by Loki includes citations and evidence sources, offering transparency and reproducibility in the fact-checking process. This feature is particularly valuable for teams conducting red-teaming exercises or evaluating AI systems for factual reliability.
The tool’s support for both open-source and commercial editions further expands its utility. While the open-source edition provides core fact-checking capabilities, the Supporter Edition offers additional features such as customizable verification data sources, enhanced user experience, and improved efficiency.