Incident Overview
The filing contained multiple citation errors and legal inaccuracies, which the firm later attributed to AI-generated hallucinations. These inaccuracies included fabricated case citations and other factual distortions introduced by the firm’s AI-assisted legal drafting workflow. The errors were identified by opposing counsel, specifically Boies Schiller Flexner LLP, prompting Sullivan & Cromwell to issue a formal correction and apology to the United States Bankruptcy Court for the Southern District of New York.
The incident occurred within the context of a high-stakes bankruptcy proceeding involving Prince Global Holdings Limited, a multinational conglomerate. The firm’s use of generative AI systems in legal drafting had not been properly reviewed or supervised, leading to the inclusion of erroneous content in a court filing.
Sullivan & Cromwell acknowledged that its AI-use and citation-review policies were not followed during the preparation of the filing. The firm’s remedial response included a formal apology to the judge presiding over the case, as well as an internal review of its AI policies to prevent future occurrences.
The harms resulting from this incident were primarily reputational and related to the dissemination of misinformation. The fabricated citations and legal inaccuracies risked undermining the integrity of the bankruptcy proceedings and could have misled the court, opposing parties, and other stakeholders involved in the case. The firm’s clients, insolvency practitioners, court-appointed liquidators, and parties to the Prince Global Holdings proceedings were all potentially affected by the errors.
This incident is classified as a variant of broader AI-related legal drafting errors, where generative AI systems produce hallucinations—fabricated or distorted information presented as factual.
Possible Root Cause
The primary root cause of the incident was the failure to adhere to Sullivan & Cromwell’s established AI-use and citation-review protocols. The firm’s AI-assisted legal drafting workflow relied on generative AI systems, which are known to produce hallucinations—outputs that appear authoritative but are factually incorrect or fabricated. In this case, the AI system generated citations and legal arguments that did not correspond to actual case law or legal precedents.
The incident highlights systemic risks associated with the deployment of AI in high-stakes legal environments. The firm’s internal policies were designed to mitigate such risks, but they were not enforced during the preparation of the emergency motion.
Generative AI systems, particularly large language models, are trained on vast datasets but lack the contextual understanding and critical reasoning required for legal analysis. When used without proper safeguards, these systems can produce plausible-sounding but incorrect legal citations, arguments, or factual claims. In this instance, the AI’s hallucinations were not detected during the drafting or review stages, leading to their inclusion in a formal court filing.
The incident also raises questions about the training and supervision of legal professionals using AI tools, as well as the need for standardized protocols in AI-assisted legal workflows.
Takeaways
This incident serves as a cautionary example for law firms and legal professionals regarding the responsible integration of AI tools into their workflows. The case demonstrates that even elite institutions are not immune to the risks posed by AI hallucinations, particularly when AI systems are deployed without adequate oversight or adherence to established policies.
One key takeaway is the critical importance of human review and verification in AI-assisted legal drafting. Despite the efficiency gains promised by AI tools, their outputs must be rigorously cross-checked against primary sources, legal databases, and peer-reviewed materials.
The firm’s response to the incident—including the correction of the filing, the apology to the court, and the internal review of AI policies—sets a precedent for accountability in AI-related legal errors. However, the incident also reveals gaps in current industry practices, particularly in the areas of AI training, policy enforcement, and stakeholder communication.
For the broader AI security community, this case highlights the need for standardized frameworks and best practices in AI-assisted legal workflows. Organizations such as the OECD’s AI Incidents and Hazards Monitor (AIM) play a role in tracking and classifying such incidents to inform policy development and risk mitigation strategies.
The harms identified in this incident—misinformation and reputational damage—are not unique to Sullivan & Cromwell but reflect broader challenges in AI adoption across industries. The dissemination of fabricated legal citations can erode confidence in judicial processes and undermine the credibility of legal institutions. Moving forward, firms must invest in AI literacy programs, develop clearer guidelines for AI use, and foster a culture of accountability to prevent similar errors.
Sources
Incident summary from the AI Incident Database, used under CC BY-SA 4.0.