Gen AI Boosts Productivity, But Can't Turn Novices Into Experts

Generative AI is helping employees stretch beyond their usual roles, easing the learning curve for attempting new tasks, but research shows it still can’t turn novices into experts.

“AI makes you feel like you can do anything. But can you do [a task] as well as people whose job it is?” says Harvard Business School Associate Professor Iavor Bojinov. “That’s what we set out to find out.”

In a study involving 78 workers at IG Group, a global derivatives trader, Bojinov and colleagues found that artificial intelligence could help marketing specialists and software developers perform the work of web analysts who write investing articles for the company’s website—but only to a point. The groups used AI equally well to organize ideas, and the marketing staff deployed AI to help them write almost as well as the web analysts. But the technology specialists—who weren’t trained in creating content—lagged behind their marketing and web colleagues by 13% when the researchers analyzed their articles for clarity and competence.

As businesses increasingly fold AI into their day-to-day tasks, the research offers insights into where AI shines most—and where it falls short: Generative AI is more useful for conceiving ideas than executing them. And AI can help people with some skills accomplish unfamiliar tasks, but the technology essentially hits a wall when people lack sufficient expertise.

The results could help inform how organizations design jobs and recruit talent as AI impacts roles, says Bojinov. He cowrote the working paper, “The GenAI Wall Effect: Examining the Limits to Horizontal Expertise Transfer Between Occupational Insiders and Outsiders,” with Assistant Professor Edward McFowland III, research associate Annika Hildebrandt, and doctoral student Matthew DosSantos DiSorbo of HBS, along with Assistant Professor Arvind Karunakaran and post-doctoral researcher Luca Vendraminelli of Stanford University.

An employee’s expertise matters

To explore AI-driven knowledge transfer across occupations, the researchers organized participants into three groups:

• Occupational insiders: 12 web analysts

• Adjacent outsiders: 26 marketing specialists

• Distant outsiders: 40 technology specialists, such as software developers and data scientists

To determine whether the technology and marketing specialists, with the help of AI, could match the performance of web analysts in organizing and writing articles on investing topics for the IG website, the researchers worked with the AI team at IG, based in the UK, to help design the experiment in 2024.

The researchers evaluated articles for clarity and writing competence based on a scoring chart provided by IG’s head of research. They also conducted interviews with participants before and after writing articles to understand how they approached the tasks.

Articles written by technology specialists with AI earned an average grade of 3.42 on a 5-point scale. That was 0.5 points, or about 13%, below the 3.96 average grade for the web analysts and the 3.92 average grade for the marketing specialists.

‘Knowledge distance’ is key

What really matters in terms of reaping AI’s benefits, the study suggests, is the employee’s “knowledge distance” from the task. For example, someone trained in one woodwind instrument could more easily pick up another woodwind instrument, while applying those same skills to a string instrument would be far more challenging.

Marketing specialists and web analysts had similar expertise profiles, which helped them outperform technical specialists, the authors say. While both groups used the same AI model, the marketing specialists “can more effectively find the information they need and then use the information to fill the necessary gaps in order to create quality content,” McFowland says. In contrast, “the folks who are too far away from the domain experts lack either sufficient understanding of the necessary information or lack the skills to use it effectively and therefore cannot match the expert’s level of quality.”

The researchers describe this as an issue of transferability: skills that are adjacent transfer more readily with AI’s help, while distant skills do not. “In other words, if you have the appropriate domain expertise,” McFowland says, “the work suggests that there are settings where AI can now allow you to produce knowledge-work outputs of similar quality as someone who spent years doing the job, perfecting their ability to execute the production process.”

AI helps with conceiving ideas

While AI couldn’t help technical specialists write at the same level as web analysts, it did have a measurable impact at an earlier phase of the article-creation process. IG personnel first spent time organizing their subject material by creating outlines and conducting other baseline research, which participants referred to as article “conceptualization.”

For those tasks, AI helped the technical and marketing specialists perform as well as the web analysts, with average grades of 4.05, 4.18, and 4.12 out of 5, respectively. Why was AI able to help more with conceiving ideas?

McFowland says it’s because conceptualization requires a structured way of thinking that’s more familiar to the technical specialists, who tend to follow a template for organizing the information needed to write the articles. Writing, on the other hand, is a less structured and more creative task, he says.

The findings suggest that generative AI helps everyone, including novices, more with conceptualization tasks, such as generating ideas and framing problems. However, the technology struggles to help novices with executing tasks, including detailed implementation and hands-on problem-solving, when they lack the necessary experience.

“When a task’s knowledge requirements are more abstract and codifiable, AI assistance goes a long way; when tasks involve concrete application and context-bound nuances, the outsider remains at a disadvantage because they lack the lived experience to interpret and enact the AI’s advice,” the researchers write. “GenAI can provide the map, but navigating the terrain is another matter.”

AI saved everyone time

While the quality of the results varied depending on the person’s occupation, AI delivered substantial productivity gains across the board. The team found:

• Conceiving articles took nearly two-thirds less time. Participants completed the conceptualization phase in 23 minutes on average using AI, compared to about 63 minutes without it.

• Writing took nearly three-quarters less time. The writing phase took 22 minutes with AI, compared to 87 minutes without it.

Where AI is most useful

The results offer guidance for companies trying to determine when AI is most useful and where its reliability breaks down. The researchers say the findings show that AI can:

Reduce training time

With AI help, data scientists could transition to other roles, such as a marketing analyst or financial analyst, within the same organization with significantly less retraining.

Shrink learning curves

“You can have flatter organizations where the learning curves for new tasks, such as SEO optimization, become much shorter and almost disappear,” says Bojinov.

Speed brainstorming

While AI may struggle with execution in some cases, it’s helpful across the board for ideation. Companies should lean into AI to shorten the time it takes to outline projects.

Image created with assets from AdobeStock.