When AI Gives Advice, Employees Rarely Ask Why

Unlike students taking a high school math exam, black-box AI systems aren’t required to show their work when answering high-stakes questions. And some decision-makers may prefer it that way—especially when their money and morals are at stake.

In an AI-assisted loan-approval experiment, participants acting as lending officers reviewing real $10,000 loan requests often chose not to know why an AI system flagged one borrower as riskier than another, even though the explanation could have revealed whether race or gender influenced the decision, research by Harvard Business School Assistant Professor Alex Chan shows.

Participants were most likely to skip explanations when their bonuses depended on loan repayment, choosing to follow the algorithm’s recommendations rather than examine whether its decisions were biased. The findings suggest that people often actively avoid seeking additional information because knowing more might complicate their decision or create moral discomfort.

As companies race to embed AI in a variety of decision-making processes, from hiring and credit approval to medical testing and judicial proceedings, pressure is mounting to ensure those systems are fair, transparent, and trustworthy. That scrutiny has fueled growing interest in “explainable AI,” which is focused on showing users the reasoning behind AI-generated responses rather than simply delivering answers.

But Chan’s research challenges a common assumption: that people naturally want more transparency from AI systems. His working paper, “Preference for Explanations: Case of Explainable AI,” updated in February, finds that in some cases, people may not want more information if it makes their decisions harder or more uncomfortable.

“Humans interacting with AI are not perfectly rational Bayesian agents,” Chan says. “They are strategic, motivated, and sometimes willfully ignorant.”

How participants evaluated loan requests

To test how people interact with AI explanations, Chan recruited 2,512 participants online and asked each to review pairs of $10,000 loan requests to a private US lender from unemployed individuals who needed money for living expenses.

Participants viewed basic demographic information about prospective borrowers, including income, race, gender, and family size, as well as an AI-generated prediction of default risk, then had the option to view an explanation for that prediction.

In each pair of loan requests, the algorithm classified one borrower as low risk, with less than a 10% chance of default, and the other as high-risk, at greater than 90%. The participants earned higher compensation when the borrower repaid the loan they selected.

Participants acting as loan officers could choose to:

1. Override the algorithm and approve both loans.

2. Follow the algorithm’s recommendation on which loan to fund.

3. Choose whether to view an explanation of the factors AI used to make its prediction before deciding.

Sometimes, people don’t want to know

Participants sought out AI’s risk predictions—but many preferred not to know why they were made, especially when it might impact their own bonus. Specifically, Chan found:

• Participants wanted the advice, but not always the reasoning behind it. Some 80% of participants wanted to see borrowers’ risk scores, compared to about 46% who sought the explanations behind the scores.

• Financial incentives dampened the desire for transparency. When participants’ compensation depended on loan repayments, they were nearly 20% more likely to decline explanations than those slated to receive a flat fee.

• Potential bias led people to avoid explanations even more. When participants were told an explanation might indicate that race or gender had influenced the AI recommendation, avoidance rates rose by more than 10 percentage points, to 23%.

• Viewing explanations spurred people to challenge AI. When participants chose to view explanations, they were about six percentage points more likely to override the AI’s recommendation and approve both loans.

Ultimately, people showed “simultaneous information-seeking and information-avoiding behavior,” Chan says. Some participants chose not to look because when they did, the explanations put them in a moral quandary and meaningfully changed their decisions, he notes.

How to use explainable AI responsibly

Chan’s research suggests that transparency alone isn’t enough and that incentives and organizational design shape when decision-makers engage with AI reasoning—and when they choose to ignore it. To make sure companies making a variety of high-stakes decisions use explainable AI responsibly, Chan recommends that business leaders:

Build oversight into AI’s decision-making processes

Government regulators are acknowledging the risks of AI. Chan points out that the European Union’s General Data Protection Regulation and Artificial Intelligence Act require disclosure of explanations for “significant algorithmic decisions.” And in 2023, the US Consumer Financial Protection Bureau reminded lenders of their legal obligation to provide borrowers with “accurate and specific” reasons for AI-assisted adverse decisions, such as credit denials.

The goal, Chan says, is not just to mandate explanations, but to make sure businesses are actually using them to shape decisions.

Create incentives for employees to engage critically with AI

Disclosure requirements for AI should focus on professional decision-makers—and go beyond what Chan calls “checkbox transparency,” given how users behave. That may require organizations to provide training and align incentives so that managers and other employees can review, document, and reflect on AI explanations, rather than simply having access to them.

“The key insight,” Chan says, “is that explainability cannot be left entirely to individual choice when individual incentives point toward willful blindness.”

Don’t forget to value human judgment

For managers, the study highlights an emerging concern in the deployment of AI: the possibility of inadvertently devaluing human judgment. That means they should encourage employees to second-guess AI’s recommendations, for instance.

“The biggest risk of AI isn’t just bad answers or lack of adoption,” Chan says. “It’s training people to stop asking why.”

Image created with asset from Unsplash/Zoha Gohar.