Study: AI model explanations fail to combat bias

Editor's Note

New data raise questions about the effectiveness of image-based AI model explanations in helping clinicians recognize systematic bias when diagnosing hospitalized patients. The findings appeared in JAMA on December 19.

Researchers looked at the diagnostic accuracy of 457 hospital physicians, nurse practitioners, and physician assistants in diagnosing patients with respiratory failure and providing treatment recommendations both with and without assistance from an AI model. Half of the clinicians received an explanation with the AI model decision, while the other half received only the AI decision with no explanation. The cases they received came from actual patients with respiratory failure, and the AI model rated whether the patient had pneumonia, heart failure, or COPD.

Clinicians using an AI model trained to make reasonably accurate predictions, but without explanations, increased their accuracy by 2.9%. When clinicians also had an explanation for the AI decision, their accuracy increased by 4.4%.

However, when researchers used a biased AI model—one that predicted a high likelihood of pneumonia if the patient was 80 years old or older, for instance—clinician accuracy decreased by 11.3%. This was the case even with explanations explicitly highlighting that the AI was looking at non-relevant information, such as bone density measurements. 

Researchers say it’s important to continue to understand the nuances of how AI tools work within the clinical environment to ensure that they are safe, secure, and consistent in improving outcomes.