MANILA, Philippines – OpenAI’s GPT, the large language model on which ChatGPT is built, increased productivity among Philippine contact center agents by 13.8% on average as measured by the number of customer issues they were able to resolve per hour. 

The study was conducted by researchers from the Stanford Digital Economy Laboratory, and the MIT Sloan School of Management, and published by US nonprofit research organization, the National Bureau of Economic Research in April 2023.

The researchers studied 3 million chats by 5,179 agents working for a Fortune 500 software firm that provides business process software. About 83% of the agents are located outside the US, primarily in the Philippines, who answer technical support questions via chat from US-based small business owners using the firm’s software. 

The AI assistant was gradually deployed among the workers, with the majority of the deployment coming in between November 2020 and February 2021. The AI assistant “monitors customer chats and provides agents with real-time suggestions for how to respond” and is “designed to augment agents, who remain responsible for the conversation and are free to ignore its suggestions.”

The researchers said that, to their knowledge, this is “the first study of the impact of generative AI when deployed at scale in the workplace” and noted that customer service sector has one of the highest adoption rates of AI technologies. 

High level of productivity improvement among lower skilled workers

The 13.8% increase in productivity was observed in three components: 

• Agents took a shorter time in handling an individual chat;
• Agents were able to increase the number of chats they were able to handle per hour;
• Agents were able to slightly increase the number of chats that were considered successfully resolved. 

The productivity improvement was significantly more pronounced in less skilled and less experienced agents, while higher skilled agents and those with more experience showed less of an improvement. 

The research team also categorized agents by skill level “based on their average call efficiency, resolution rate, and surveyed customer satisfaction in the quarter prior to the adoption of the AI system.” Those in the lowest tier showed the greatest improvement in productivity at 35%. 

Must Read

AI eating into ‘contract cheating’ pie, leaving Kenyan ghostwriters at a loss – report

The AI assistant also helped workers improve much faster, with researchers noting that agents with two months of experience who used the assistant were as productive as agents with 6 months of experience who did not use the assistant. 

Why does this happen? The researchers theorized that the higher skilled workers have less to gain “because AI recommendations capture the potentially tacit knowledge embodied in their own behaviors.” 

In other words, AI’s recommendations, trained using data from experienced workers, are already embedded in the wisdom and experience of higher skilled, longer tenured workers. They found “suggestive evidence that AI assistance leads lower skill agents to communicate more like high-skill agents,” which they posit may help lower attrition rates in the industry. 

“This suggests that an AI system may be able to more effectively share knowledge across workers both because it may capture tacit knowledge that was previously difficult for managers to articulate and because it can provide more real-time recommendations than a busy manager,” the researchers wrote. 

Observing chat message sentiments, they also noted an improvement in the way that customers treated agents, who learned the job faster with the aid of the AI assistant.

“Our overall findings demonstrate that generative AI working alongside humans can have a significant positive impact on the productivity and retention of individual workers,” the researchers wrote. 

Study not an indicator of future job trends

On the study’s weaknesses, the researchers make it clear that their findings do not provide how AI may affect employment demands from companies. They said, “Our results do not capture potential longer-term impacts on skill demand, job design, wages, or customer demand.”

For example, more effective technical support could accelerate the trend towards contact center agents taking on more complex customer responsibilities, increasing aggregate demand even if agents become more productive.” 

Regarding the efficacy of AI in such a work setting, they said this industry “has a relatively stable product and set of technical support questions.

“In areas where the product or environment is changing rapidly, the relative value of AI recommendations – trained on historical data – may be different.” 

Finally, like creators rightfully demanding compensation for their works that train generative AI, the researchers said that their findings raise questions on whether high-skilled workers should also be compensated for the high quality training data they generate for the AI systems with their quality work.

“High-skill workers, in particular, play an important role in [AI] model development but see smaller direct benefits in terms of improving their own productivity.” – Rappler.com