When AI Chatbots Help People Act More Human

Featuring Shunyuan Zhang and Das Narayandas. By Ben Rand on May 27, 2025.

An analysis of more than 250,000 chat conversations reveals the potential for AI chatbots to improve customer service. Research by Shunyuan Zhang and Das Narayandas.

Laptop with chat bubble on screen, with human and robot hands on the keyboard

Can an AI chatbot help customer service representatives, well, be more human? Maybe so, suggests a new study that adds to a chorus of findings on how businesses can best tap the new technology.

Researchers at Harvard Business School analyzed a year’s worth of online chat conversations between a meal delivery company and its customers to determine how AI affects response times and customer service.

According to their study, AI helped human agents respond to chats some 20 percent faster—improving performance even more for less experienced agents. And, the technology helped humans reply with more empathy and thoroughness—uniquely human service strengths with the potential to positively impact revenue and profitability by making for happier customers.

You should not use AI as a one-size-fits-all solution in your business, even when you are thinking about a very specific context such as customer service.

Companies from banks to boutiques have been turning to chatbots, hoping that AI can provide faster, cheaper, and more accessible service. However, recent findings point to a growing understanding that AI currently works best as a complement to human intelligence, rather than a replacement. Although AI iterates and improves rapidly, concerns about biased large language models and misleading results suggest that leaders might need to proceed cautiously.

“You should not use AI as a one-size-fits-all solution in your business, even when you are thinking about a very specific context such as customer service,” says HBS Assistant Professor Shunyuan Zhang. “As the paper shows, there are boundaries, and the effects of AI vary across different customer intents.”

Zhang worked with Das Narayandas, the Edsel Bryant Ford Professor of Business Administration at HBS, on “Engaging Customers with AI in Online Chats: Evidence from a Randomized Field Experiment,” which is forthcoming in Management Science.

How AI improves customer service

The study focused on the delivery company’s use of an agent-based support AI tool developed by L.ai, a New York-based firm that aims to make customer service more human. Some agents received real-time response suggestions from the AI tool, while others did not.

The tool began as an AI crisis hotline designed to manage difficult conversations with empathy and appropriate suggestions. It was trained on a database of more than 3 million diverse customer-service situations to recommend responses with four basic steps: an offer to help, an apology, a validation formula, and gratitude.

The researchers designed an experiment involving 138 customer service agents at an unnamed meal delivery company. They analyzed 256,934 online chat conversations between December 2020 and November 2021 for efficiency and customer sentiment. Customer sentiment was evaluated by the AI tool based on emotional tone at the beginning and end of the interaction.

Agents who used AI-based suggestions experienced:

22%
Drop in response times, even as agent-customer interactions increased
0.45 pts
Increase in customer sentiment on a five-point scale, compared with agents without AI

Among less-experienced agents, the results were more pronounced:

70%
Decline in response time
1.63 pts
Increase in customer sentiment

When AI works best—and when it doesn’t

While AI assistance generally led to improved customer sentiment, some types of inquiries showed more benefit than others.

The study found that AI improved sentiment when customers asked to cancel their subscriptions. The researchers think this could be because AI recommended alternative subscription options.

However, the improvement was less substantial when customers contacted the company with repeat complaints, which suggests that these matters transcended AI’s capabilities.

When customers were transferred to human agents after their initial engagement with AI chatbots, the technology responded so fast that some customers felt confused. In those cases, customer sentiment scores improved less—most likely, the authors say, because the transfer happened so quickly that customers thought they were still communicating with a chatbot.

“AI helped agents respond to customers more rapidly, which is a good thing,” Zhang says. “But when it’s too fast, customers kind of wonder, ‘is this still AI?’”

New ways to harness AI

Managers implementing AI tools for employees should focus on the question of “who would benefit more?” Zhang says. The research suggests that AI is effective for:

Training new employees in service roles

Perhaps most significantly, the researchers discovered that AI support helped less-experienced agents most. The researchers estimate that agents would need almost a year and a half of work experience to produce those improvements on their own.

“Now AI can help boost newer agents along the learning curve. It can make them feel better about their handling of work on a day-to-day basis,” she says. Confident employees might be easier for companies to retain. Employee attrition in customer service roles has been a “huge problem,” she says.

Helping companies keep “freemium” customers

AI-assistance could help firms that offer freemium—free service with fewer features—or subscription-based models retain clients. That’s because AI was most useful to agents in the study when customers called to cancel subscriptions.

After all, keeping non-paying customers, through positive service experiences, means that the company can try to convert them into paying subscribers later.

Image by HBSWK with assets from AdobeStock.

Engaging Customers with AI in Online Chats: Evidence from a Randomized Field Experiment

Zhang, Shunyuan, and Das Narayandas. "Engaging Customers with AI in Online Chats: Evidence from a Randomized Field Experiment." Management Science (forthcoming).

Access the article

Have feedback for us?

Have questions or feedback for us? Send us a note to hbswk@hbs.edu.

Article By:

Ben Rand, Writer

Illustrated portrait of Ben Rand, a writer at Harvard Business School's Working Knowledge online publication

Topics:

Artificial IntelligenceCustomer SatisfactionData and TechnologyPerformance ImprovementTechnology Adoption

Latest from HBS faculty experts

Expertly curated insights, precisely tailored to address the challenges you are tackling today.

More about Working Knowledge