AI chatbots like ChatGPT can be dangerous for doctors as well as patients, as …, warns MIT Research

A new study from MIT researchers reveals that Large Language Models (LLMs) used for medical treatment recommendations can be swayed by nonclinical factors in patient messages, such as typos, extra spaces, missing gender markers, or informal and dramatic language. These stylistic quirks can lead the models to mistakenly advise patients to self-manage serious health conditions instead of seeking medical care. The inconsistencies caused by nonclinical language become even more pronounced in conversational settings where an LLM interacts with a patient, which is a common use case for patient-facing chatbots.

Published ahead of the ACM Conference on Fairness, Accountability, and Transparency, the research shows a 7-9% increase in self-management recommendations when patient messages are altered with such variations.