Incident Overview
The document included at least six fictitious academic references that could not be verified by experts or journal editors. These references appeared to be fabricated, a phenomenon widely attributed to AI hallucinations—where large language models generate plausible but incorrect citations. The draft was later withdrawn after academics and journalists flagged the discrepancies. This incident exposed a critical failure in the use of AI-assisted tools in policymaking, particularly in a document intended to govern the ethical and technical standards of AI deployment in the country.
A similar issue emerged shortly after with the Department of Home Affairs (DHA). Its Cabinet-approved Revised White Paper on Citizenship, Immigration, and Refugee Protection appended a standalone list of references that also contained fictitious entries. These references were not cited within the body of the document but were included as a separate section. The DHA acknowledged the error, removed the list, suspended two senior officials, and commissioned an independent review of all policy documents produced since November 2022. The review aimed to assess whether similar AI-generated inaccuracies had occurred in other government publications.
Both incidents underscore a broader challenge: the integration of AI tools into high-stakes policymaking processes without adequate safeguards. The errors did not alter the substantive policy positions in either document, but they eroded public trust in the government’s ability to produce reliable, evidence-based policy. The use of AI in drafting official documents—particularly in a context where AI governance is the subject—highlighted the irony of relying on the very technology being regulated to produce foundational policy frameworks.
Four officials across the two departments were suspended pending investigations, though specific details about their roles or the extent of their involvement remain limited. The DCDT initially defended the draft policy, arguing that the fictitious references did not affect its core content, but the withdrawal of the document signaled recognition of the reputational and epistemic harm caused by the errors.
The term ‘hallucination’ was widely used by experts and officials to describe the fabricated references, framing the issue as a direct consequence of unchecked reliance on generative AI systems. The incidents were also linked to a broader classification of AI risks, particularly those related to misinformation and epistemic integrity—the reliability of information used to inform policy and public discourse.
The South African government’s response included corrective actions such as document withdrawals, official suspensions, and independent audits. However, the incidents raised questions about the preparedness of public institutions to adopt AI tools responsibly. They also served as a case study in how AI systems, when deployed without rigorous oversight, can introduce errors that undermine the credibility of institutional processes.
Possible Root Cause
The primary root cause of the fictitious references in both the National AI Policy draft and the Home Affairs White Paper appears to be the uncritical use of large language models (LLMs) and generative AI tools in the drafting process. Experts and officials attributed the errors to AI hallucinations—a known limitation of generative models where they produce plausible but incorrect or fabricated information, including citations, statistics, and source attributions.
The incidents suggest that the departments involved did not implement sufficient safeguards to verify the accuracy of AI-generated content.
The use of AI tools was likely intended to streamline the drafting process, but without adequate oversight, it introduced risks that outweighed the potential efficiencies.
In this case, the use of AI in drafting a policy on AI governance created a paradox: the very tool intended to assist in regulation became a source of regulatory failure.
While the exact AI systems used by the departments remain unidentified, the pattern of errors aligns with known behaviors of commercial LLMs, which are prone to generating fabricated references when prompted to assist in research or drafting. The lack of transparency around the specific models or tools employed further complicates efforts to attribute responsibility or design mitigations. However, the consistent appearance of hallucinated references in both documents points to a systemic issue rather than an isolated technical glitch.
Takeaways
The incidents in South Africa’s AI policy and Home Affairs white paper highlight several critical lessons for governments and institutions adopting AI tools in high-stakes environments.
Even advanced generative models cannot replace the role of subject-matter experts in verifying facts, citations, and technical accuracy. Institutions must implement robust review mechanisms, including manual checks by domain specialists, to ensure the integrity of AI-generated content.
Governments should disclose when AI systems are used in policymaking and outline the measures taken to validate their outputs. This transparency is particularly crucial in documents intended to shape regulatory frameworks, where public trust in the accuracy of information is foundational. The absence of such disclosures in South Africa’s cases exacerbated the reputational damage, as stakeholders only became aware of the errors after they were published.
Third, the errors reveal gaps in institutional preparedness for AI integration. Additionally, departments should establish clear protocols for AI use, including approval processes, documentation requirements, and contingency plans for errors.
Policymaking, legal drafting, and technical regulation are domains where precision and reliability are non-negotiable. The use of AI in these contexts should be approached with heightened caution, and institutions should prioritize tools and workflows that minimize the risk of fabricated or misleading content. This may involve preferring AI systems with demonstrated strengths in factual accuracy or supplementing them with human-in-the-loop verification.
Finally, the cases illustrate the broader implications of AI failures in governance. Misinformation and reputational harm are not abstract risks but tangible consequences of unchecked AI deployment. For South Africa, where AI adoption is still evolving, these incidents could delay progress in establishing trustworthy AI policies.
The South African government’s response—withdrawing flawed documents, suspending officials, and initiating reviews—sets a precedent for accountability. Other governments and organizations would be wise to study these incidents as a case study in the challenges of balancing AI efficiency with the demands of institutional integrity.
Sources
- incidentdatabase.aihttps://incidentdatabase.ai/cite/1467/
- incidentdatabase.aihttps://incidentdatabase.ai/cite/1548/
Incident summary from the AI Incident Database, used under CC BY-SA 4.0.