Source: Chen & Chan (2024) — “Large Language Model in Creative Work: The Role of Collaboration Modality and User Expertise,” Management Science
The thing I like about this paper is that it complicates the easy story. It is not simply “AI good” or “AI bad.” It is more like: AI changes the work, and then the real problem moves somewhere less obvious.
Here’s a finding I keep thinking about: two groups of people use the same AI tool for the same creative task. One group sees their output quality go up. The difference isn’t the AI — it’s how they used it. This paper runs a controlled experiment where expert and non-expert users write advertising copy with and without LLM assistance. The twist is that “with LLM assistance” is split into two distinct collaboration modes: using the AI as a ghostwriter, where the AI generates the content and the human refines it, versus using the AI as a sounding board, where the human writes first and the AI critiques it. Performance is measured by actual click-through rates on real social media platforms — not subjective quality scores, but real market behaviour. The results split cleanly along two dimensions. For non-experts: using the AI as a sounding board improves ad quality significantly. Having the AI critique their work apparently helps them identify weaknesses they couldn’t see themselves and produce better output. But using the AI as a ghostwriter — letting it write the first draft — doesn’t help as much.
In plain English, that is why the result matters beyond the chart. It changes where people should look, what they should question, and which comfortable assumption probably needs to be retired.
My takeaway: AI rarely removes the hard part. It relocates it. The task gets faster, and then humans still have to decide what matters, what to trust, and what could quietly go wrong.