Incident Overview
On October 21, 2025, the United States Court of Appeals for the Sixth Circuit identified false quotations generated by Thomson Reuters’ Westlaw CoCounsel in the case United States v. The incident involved court-appointed attorney Steven N. Howe, who used the AI-powered legal research tool to draft appellate briefs. The court later determined that these briefs contained fabricated legal citations and misleading descriptions of precedent, undermining the integrity of the appeal process. The false quotations were not isolated errors but part of a broader issue affecting the accuracy of AI-assisted legal research tools.
The case is significant because Westlaw CoCounsel is a retrieval-grounded legal AI system specifically designed to mitigate hallucinations—a common challenge in generative AI tools. Its reliance on structured legal databases and citation accuracy claims made the incident particularly notable. The court’s ruling highlighted that even specialised AI tools, when integrated into high-stakes legal proceedings, can produce unreliable outputs with serious consequences.
Farris, the defendant, and the restarting of the briefing process. Howe, who had used CoCounsel to draft the initial filings, faced professional repercussions, including the loss of compensation and potential disciplinary action by the bar. The Sixth Circuit’s opinion, issued on April 3, 2026, explicitly documented the use of CoCounsel and the resulting harm to judicial integrity and epistemic reliability in legal proceedings.
The incident was first documented in the AI Incident Database (AIID) on July 6, 2026, under Incident ID 1572. The filing date of Howe’s principal appellate brief, October 21, 2025, serves as the key reference point for the timeline of this event. The court’s decision underscored the risks of AI-generated legal content, even when deployed in tools marketed for precision and accuracy.
The false quotations were not merely errors of omission but actively misleading representations of legal authorities. This raised concerns about the broader reliability of AI-assisted legal research systems, particularly in contexts where precision is critical. The case serves as a cautionary example for legal professionals and AI developers alike, illustrating that retrieval-grounded systems are not immune to generating fabricated or distorted legal content.
Possible Root Cause
Farris appears to stem from the limitations of retrieval-augmented generation (RAG) systems, even when applied to highly structured legal datasets. Westlaw CoCounsel, like other retrieval-grounded AI tools, relies on retrieving relevant legal sources from its database and generating responses based on those inputs. However, the incident suggests that the system may have either misinterpreted the retrieved sources or introduced synthetic content that did not accurately reflect the original materials.
In this case, the court’s opinion indicated that CoCounsel generated citations that were not verbatim matches to any existing legal authority. This suggests a failure in either the retrieval process—where irrelevant or misleading sources were selected—or the generation process, where the AI fabricated content to fill gaps in its knowledge base.
Another contributing factor may be the tool’s design assumptions about user oversight. Legal professionals using CoCounsel likely assumed that the AI’s outputs were fact-checked and reliable, given its marketing emphasis on citation accuracy. However, the incident reveals that even when AI tools are deployed in professional settings with high stakes, they can produce outputs that require independent verification. The lack of built-in safeguards against hallucination in retrieval-grounded systems remains a critical vulnerability, particularly in domains where precision is non-negotiable.
The case also highlights the challenges of integrating AI into legal workflows without adequate human review. Howe’s reliance on CoCounsel for drafting appellate briefs suggests a potential overconfidence in the tool’s capabilities. While AI can assist in legal research by surfacing relevant cases and summarising arguments, the final responsibility for accuracy lies with the attorney. The incident underscores the need for legal professionals to treat AI-generated content as provisional and subject to rigorous scrutiny, rather than as definitive authority.
Legal databases are vast and constantly evolving, with nuances in case law that even advanced retrieval systems may struggle to capture accurately. If the AI’s training data or retrieval mechanism was outdated or incomplete, it could have led to the generation of citations that appeared plausible but were factually incorrect. This aligns with broader observations about the limitations of RAG systems in maintaining real-time accuracy across dynamic knowledge domains.
Takeaways
First, it demonstrates that retrieval-grounded AI systems are not inherently immune to generating false or misleading content, despite their reliance on structured data sources. The case challenges the assumption that such tools inherently solve the problem of hallucination, particularly in high-stakes environments like legal proceedings where precision is paramount.
Legal professionals cannot treat AI-generated outputs as definitive without independent verification. The consequences of unchecked reliance on AI tools—such as procedural delays, loss of professional standing, and erosion of judicial trust—are significant and warrant careful consideration. Firms and practitioners must implement robust review processes to ensure the accuracy of AI-generated legal content before submission.
Third, the case highlights the importance of transparency in AI tool documentation and marketing. Westlaw CoCounsel was promoted as a solution to hallucination in legal AI, yet its failure in Farris exposes a gap between its advertised capabilities and real-world performance. Developers must clearly communicate the limitations of their tools, including the potential for errors in retrieval and generation, to manage user expectations and mitigate risks.
Even tools that perform well in controlled environments may fail under the complexities of actual legal practice. The legal community would benefit from establishing standards for AI tool validation, particularly in domains where errors carry severe consequences.
The harm caused by false quotations in Farris was not just technical but also epistemic—undermining trust in legal processes and potentially affecting judicial outcomes. Organizations deploying AI in legal settings must prioritise not only the accuracy of outputs but also the integrity of the systems that produce them.
For AI developers, the incident is a call to refine retrieval mechanisms and generation processes to reduce the risk of fabricated content. For legal professionals, it is a reminder that AI tools are assistants, not replacements, and that their outputs must always be cross-verified against primary sources.
The broader implications of this incident extend to the regulation and governance of AI in legal practice. The Farris case may prompt discussions on how courts and regulatory bodies can adapt to the challenges posed by AI-generated legal content, ensuring that innovation does not come at the cost of accuracy or fairness.
Sources
Incident summary from the AI Incident Database, used under CC BY-SA 4.0.