Become a Member

Regulation TrackerLaws, orders and treaties read for their security duties and clocks Standards & FrameworksAttack taxonomies, control catalogues and frontier frameworks AI Incident DatabaseThe security incident record, with mechanism and lesson Research Paper LibraryThe papers that define the field, summarised OSS DirectoryRed teaming, guardrail, supply chain and evaluation tools Case StudiesProgrammes built, with measured outcomes AI Vulnerability TrackerCVEs and advisories for the components AI systems are built from

Latest in the tracker

New York Stop Addictive Feeds Exploitation (SAFE) for Kids Act and implementing rulesNew York's law restricting algorithmic feeds for minors takes effect on 25 January 2027 under final rules published in July 2026, with certified age assurance and $5,000-per-violation penalties.

Knowledge Base Regulation Tracker Standards & Frameworks OSS Directory AI Incident Database Research Paper Library Case Studies Society Articles News Jobs About Charter Become a Member

Home/AI Incident Database/Widely used health algorithm systematically under-refers Black patients for extra care

Widely used health algorithm systematically under-refers Black patients for extra care

Obermeyer and colleagues showed in Science that a commercial care-management algorithm ranked healthier white patients above sicker Black patients because it predicted cost rather than illness.

Severity: Highoptumhealthcare biasproxy discrimination

At a glance

Organisation: Optum (UnitedHealth); US hospitals
Industry: Healthcare
Country or region: United States
Occurred: 1 Jan 2019
Disclosed: 25 Oct 2019
Type of harm: Bias and discrimination, Wrongful denial of services or benefits, Medical and safety-critical error

All details 13

Kind of system: Automated decision-making systems
Severity: High
Confirmation: Demonstrated by researchers
Technique: Healthcare cost used as a proxy for need, encoding lower spending on Black patients
Primary root cause: Bias in training data
Scale: Algorithm applied to about 200 million people per year; fixing it would raise Black patients' share of extra-care referrals from 17.7% to 46.5%
Outcome: Optum worked with the researchers to reduce bias; New York regulators investigated and the study became the leading citation for proxy discrimination in healthcare AI.

Sources

science.orghttps://www.science.org/doi/10.1126/science.aax2342

Contribute

Spot an error, or know a record we should add? Members can propose additions and submit corrections.

Submit incident Report a correction or request removal