Joint work with Jason Chan.

This paper is currently under minor revision at Management Science.

Abstract

Since the launch of ChatGPT in Dec 2022, Large Language Models (LLMs) are rapidly being adopted by businesses to assist users in a wide range of open-ended tasks, including ones that require creativity. While the flexibility of LLM unlocked new ways of human-AI collaboration, it remains uncertain whether LLMs can truly enhance business outcomes. To examine the effects of human-LLM collaboration on business outcomes, we conducted an experiment where we t asked expert and non-expert users to write an ad copy with and without the assistance of LLMs. Here, we investigate and compare two ways of working with LLMs:

1. using LLMs as “ghostwriters,” which assume the main role of content generation tasks and
2. using LLMs as “sounding boards,” to provide feedback based on human-created content.

We measure the quality of the ads using the number of clicks generated by the created ads on major social media platforms. Our results show that different collaboration modalities can result in very different outcomes for different user types. Using LLMs as sounding boards enhances the quality of the resultant ad copies, especially for non-experts. However, using LLMs as ghostwriters did not provide significant benefits and is in fact detrimental to expert users. We rely on textual analyses to understand the mechanisms, and learned that using LLMs as ghostwriters produces an anchoring effect which leads to lower-quality ads. On the other hand, using LLMs as sounding boards helped non-experts achieve ad content with low semantic divergence to content produced by experts, thereby closing the gap between the two types of users.