Does a Popular Bias Test Actually Measure Bias?

For more than two decades, researchers and businesses have used the Implicit Association Test to tease out hidden biases that may shape how people make decisions in workplaces, law enforcement agencies, hospitals, schools, and other organizations.

But a study now suggests that the test may be widely misinterpreted as a direct way of measuring a person’s unconscious bias. It was previously assumed that people took longer to answer questions, in some cases because it took them time to process unfamiliar associations, such as between the concepts “white” and “bad,” which might reveal some bias. But in studying the Implicit Association Test (IAT), a team including Harvard Business School Associate Professor Amit Goldenberg analyzed data from more than 115,000 test-takers and concluded that people may take longer primarily because they want to respond cautiously to avoid making mistakes.

Companies often use the IAT as part of workplace diversity and inclusion efforts, particularly to help employees recognize and reflect on potential implicit bias against people based on race, gender, age, and disability. Organizations may also use the test to assess whether interventions aimed at preventing bias in hiring and promotion decisions are working. Goldenberg’s research results question whether the test actually measures bias in the way it was intended.

In using the IAT, "they're getting an accurate estimation of something,” says Goldenberg. “I just don't know if it's bias as it was defined originally.”

The research, “Challenging the Mechanism for the Implicit Association Test,” appeared in the June issue of Nature Human Behaviour. Goldenberg coauthored the article with Kyle J. LaFollette, a principal researcher and statistical consultant at the University of Chicago, and Professor Heath A. Demaree and researcher Doroteja Rubez, both of Case Western Reserve University.

Testing for hidden bias

Social scientists have used the IAT since the mid-1990s to detect unconscious biases—attitudes or stereotypes that affect our actions or decisions without our knowledge. During the test, participants associate categories such as race and gender with positive or negative concepts. For instance, participants may associate images of Black and White faces with certain good or bad words. The test has long been the subject of debate among researchers over whether its scores indicate genuine unconscious bias, or something else entirely.

To help answer that question, Goldenberg and his team analyzed the results of 115,601 unique IATs completed between December 2007 and June 2012 through Project Implicit, an international online research platform. The dataset included 39 different types of IATs, each with at least 2,673 participants.

The test’s signature metric, the D-score, typically produces a single, summary assessment. In this case, researchers instead analyzed each individual data point, using computational modeling to break the person’s reaction time into different variables. In doing so, their analysis discovered why someone responded a certain way, rather than evaluating only how fast the person responded.

What’s driving the test results?

The study identified two variables as factors in IAT results, both of which can be implicit—meaning done without awareness:

• Decision ease: a measure of comfort in making mental associations between concepts.

• Response caution: a tendency to slow down and take extra care when confronted with ideas that feel uncomfortable.

When the researchers compared the two, response caution proved to be the stronger factor, having roughly 60% more influence over the D-score than decision ease. The same cautious effect “is seen across studies, across people in a variety of contexts and different time points,” Goldenberg says.

“Does the fact that you're slowing down suggest you have a bias?” he asks. “This very much depends on what we call bias.”

What the findings mean for organizations

Based on the results, Goldenberg advises that managers exercise caution in using IAT as an evaluation tool for their inclusion programs.

For example, a company might give employees a test as part of a bias training program and compare scores before and after the training. But they should be wary of concluding that if scores change, the intervention impacted people’s biases, since other factors may be at play.

A new view of the test

Goldenberg stresses that the study doesn’t question the existence of unconscious bias. It’s certainly real, he says, and social scientists should continue working to find better ways to measure it.

In the future, he says, someone will likely design a new test that separates response caution and decision ease, enabling leaders to get closer to the goal of identifying implicit biases and their hidden role in decision-making.

Until then, it’s important for organizations that use the IAT to adjust the meaning they attach to the results.

"There's enough explicit bias in the world that you can filter employees based on that,” Goldenberg says. “Let's start with that, and then we'll move to implicit bias once the tools are better."

Image: Ariana Cohen-Halberstam