Incident Overview
In April 2025, the Department of Government Efficiency (DOGE) under the Trump administration reportedly used ChatGPT to evaluate and flag National Endowment for the Humanities (NEH) grants for cancellation. Based on these unvetted AI outputs, DOGE compiled a list of grants to terminate, leading to the cancellation of over a thousand awards with minimal human oversight. This decision disrupted humanities organizations and projects nationwide, as funding was clawed back without adequate review.
The incident became a reference case for AI-driven administrative decisions, highlighting risks associated with relying on unvalidated AI systems in high-stakes decision-making processes. Litigation by affected grantees followed, resulting in the reinstatement of some awards. Court documents revealed specific examples, such as a $349,000 grant for replacing a museum’s HVAC system being canceled after ChatGPT flagged it as DEI-related, despite the project having no direct connection to DEI initiatives.
The use of ChatGPT in this context was framed as a response to executive orders aimed at reducing DEI initiatives. DOGE employees, including Nate Cavanaugh and Justin Fox, allegedly fed grant descriptions into the model using prompts like 'Does the following relate at all to DEI?' The outputs were then used to justify cancellations, raising concerns about the accuracy, fairness, and transparency of AI-assisted decision-making in public administration.
The incident underscores broader risks in AI system deployment, particularly when models are used without rigorous validation, human oversight, or clear guidelines for interpretation.
Possible Root Cause
The primary root cause of this incident appears to be the reliance on unvetted and unverified outputs from ChatGPT, a large language model not designed or validated for administrative decision-making. DOGE staff used the model to screen grants without establishing protocols for accuracy, bias, or contextual understanding.
ChatGPT’s limitations in interpreting nuanced or context-dependent information contributed to errors. The model’s outputs lacked the domain-specific expertise required to assess complex humanities projects, leading to misclassifications and unjustified cancellations.
The decision to automate grant evaluations using an off-the-shelf AI model without safeguards created an environment where flawed decisions could have significant real-world consequences.
The involvement of DOGE employees in designing and executing this process without oversight suggests a failure in governance. The agency’s approach treated ChatGPT as a substitute for human judgment, rather than as an assistive tool requiring careful calibration and review. This lack of structural oversight allowed the incident to escalate, with widespread disruptions to grantees and organizations.
Takeaways
This incident serves as a cautionary example of the dangers of deploying AI systems in high-stakes administrative functions without proper safeguards. Key takeaways include the necessity of human oversight in AI-assisted decision-making, particularly in contexts where errors can lead to wrongful denials or misallocations of resources.
In this case, ChatGPT’s outputs were treated as authoritative, despite its lack of domain-specific training or accountability mechanisms.
DOGE’s use of ChatGPT lacked transparency, making it difficult to audit or challenge the decisions made. Future implementations should require detailed logs of AI interactions, explanations for outputs, and processes for human review and appeal. This would help mitigate risks of misinformation or fabricated claims, as seen in this incident.
The harm caused by this incident extended beyond individual grantees to the broader epistemic integrity of public institutions.
Legal and reputational consequences followed the incident, including litigation and media scrutiny. Grantees challenged the cancellations, leading to some reinstatements, but the broader damage to affected organizations and the precedent set for AI in governance remained. This case demonstrates that AI risks are not merely technical but also legal, ethical, and societal in nature.
ChatGPT, like other large language models, is not intended for high-stakes decision-making without human intervention.
Finally, this case illustrates the need for interdisciplinary approaches to AI governance.
Sources
Incident summary from the AI Incident Database, used under CC BY-SA 4.0.