Subtle biases in AI can influence emergency decisions | MIT News

It’s no secret that people harbor biases—some unconsciously, perhaps, and others painfully overt. The average person might assume that computers—machines typically made of plastic, steel, glass, silicon, and various metals—are free of bias. While that assumption may be true for computer hardware, the same is not always true for computer software, which is programmed by fallible humans and may carry data that is itself compromised in some way.

Artificial intelligence (AI) systems – especially those based on machine learning – are increasingly used in medicine for the diagnosis of specific diseases, for example, or the evaluation of X-rays. These systems are also relied upon to support decision making in other areas of healthcare. However, recent research has shown that machine learning models can encode biases against minority subgroups, and the recommendations they make can consequently reflect those same biases.

A new study by researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) and the MIT Jameel Clinic, published last month in Communication medicine, assesses the impact that discriminative AI models can have, especially for systems intended to provide advice in urgent situations. “We found that the way the advice is framed can have significant consequences,” explains the paper’s lead author, Hammaad Adam, a PhD student at MIT’s Institute for Data Systems and Society. “Fortunately, the damage caused by biased models can be limited (though not necessarily eliminated) when the advice is presented in a different way.” The other co-authors of the paper are Aparna Balagopalan and Emily Alsentzer, both PhD students, and professors Fotini Christia and Marzyeh Ghassemi.

AI models used in medicine can suffer from inaccuracies and inconsistencies, in part because the data used to train the models are often not representative of real-world environments. Different types of X-ray machines, for example, can record things differently and therefore produce different results. Moreover, models trained primarily on white people may not be as accurate when applied to other groups. The Communication medicine paper is not focused on issues of that kind, but rather addresses problems arising from biases and on ways to mitigate the adverse effects.

A group of 954 people (438 clinicians and 516 non-experts) participated in an experiment to see how AI biases could affect decision making. Participants were presented with call summaries from a fictitious crisis hotline, each involving a male individual undergoing a mental health case. The summaries contained information on whether the individual was Caucasian or African American and would also state his religion if he happened to be Muslim. A typical call summary might describe a circumstance in which an African-American man was found at home in a delirious state, indicating that “he had not used any drugs or alcohol as he is a practicing Muslim.” Study participants were instructed to call the police if they thought the patient was likely to become violent; otherwise, they were encouraged to seek medical help.

The participants were randomly divided into a control or “baseline” group plus four other groups designed to test responses under slightly different conditions. “We want to understand how biased models can affect decisions, but first we need to understand how human biases can affect the decision-making process,” notes Adam. What they found in their analysis of the baseline group was quite surprising: “In the setting we considered, human participants showed no biases. That doesn’t mean people aren’t prejudiced, but the way we conveyed information about a person’s race and religion was obviously not strong enough to elicit their prejudices.”

The other four groups in the experiment received advice that came from either a biased or unbiased model, and that advice was presented in either a “prescriptive” or a “descriptive” form. A biased model would be more likely to recommend police assistance in a situation involving an African American or Muslim person than an unbiased model. However, participants in the study did not know what kind of model their advice was coming from, or even that models providing the advice could be biased at all. Prescriptive advice spells out what a participant should do in no uncertain terms, telling them to call the police in one case or seek medical help in another. Descriptive advice is less direct: A flag is displayed to show that the AI ​​system perceives a risk of violence associated with a specific call; no flag is shown if the threat of violence is deemed low.

An important takeaway from the experiment is that participants were “highly influenced by prescriptive recommendations from a biased AI system,” the authors wrote. But they also found that “using descriptive rather than prescriptive recommendations allowed participants to retain their original, unbiased decision making.” In other words, the bias built into an AI model can be reduced by tailoring the advice delivered appropriately. Why the different outcomes depending on how advice is stated? When someone is told to do something, like call the police, it leaves little room for doubt, Adam explains. However, when the situation is simply described – classified with or without the presence of a flag – “it leaves room for a participant’s own interpretation; it enables them to be more flexible and consider the situation themselves.”

Second, the researchers found that the language models typically used to offer advice are easily biased. Language models represent a class of machine learning systems trained on text, such as the entire content of Wikipedia and other web material. When these models are “fine-tuned” by relying on a much smaller subset of data for training purposes—just 2,000 sentences, as opposed to 8 million web pages—the resulting models can easily be biased.

Third, the MIT team discovered that decision makers who are themselves unbiased can still be misled by the recommendations provided by biased models. Medical training (or the lack thereof) did not change responses in any detectable way. “Clinicians were just as influenced by biased models as were non-experts,” the authors said.

“These findings may apply to other settings,” says Adam, and are not necessarily limited to healthcare situations. When it comes to deciding which people should receive a job interview, a biased model may be more likely to reject Black applicants. The results may be different, however, if instead of telling an employer explicitly (and prescriptively) to reject this applicant, a descriptive flag is attached to the file to indicate the applicant’s “possible lack of experience” .

The implications of this work are broader than just figuring out how to deal with individuals in the midst of mental health crises, Adam maintains. “Our ultimate goal is to make sure that machine learning models are used in a fair, safe and robust way.”