Source: Chen & Chan (2024) — “Large Language Model in Creative Work: The Role of Collaboration Modality and User Expertise,” Management Science
Here’s a finding I keep thinking about: two groups of people use the same AI tool for the same creative task. One group sees their output quality go up. The other group sees it go down. The difference isn’t the AI — it’s how they used it.
This paper runs a controlled experiment where expert and non-expert users write advertising copy with and without LLM assistance. The twist is that “with LLM assistance” is split into two distinct collaboration modes: using the AI as a ghostwriter (the AI generates the content, the human refines it) versus using the AI as a sounding board (the human writes the content, the AI critiques it). Performance is measured by actual click-through rates on real social media platforms — not subjective quality scores, but real market behaviour.
The results split cleanly along two dimensions.
For non-experts: using the AI as a sounding board improves ad quality significantly. Having the AI critique their work apparently helps them identify weaknesses they couldn’t see themselves and produce better output. But using the AI as a ghostwriter — letting it write the first draft — doesn’t help as much. Non-experts who let the AI lead the creative process end up with output that’s better than their solo baseline, but not as good as when they use AI for feedback.
For experts: the pattern reverses in a striking way. Using the AI as a ghostwriter actively hurts output quality. Expert copywriters who let the AI generate the initial content end up with worse ads than experts who worked alone. The sounding board mode is more neutral for experts — it neither helps nor hurts meaningfully.
The mechanism the researchers identify is anchoring. When you start from an AI-generated draft, you anchor to it — your revisions stay within the creative space the AI has already defined. For non-experts, this anchoring is mostly helpful because the AI’s starting point is better than where they’d begin on their own. For experts, it’s harmful because it constrains them away from the creative directions that actually made their work good in the first place.
There’s something important here about the asymmetry of expertise and AI assistance. The conventional wisdom is that AI helps people who need help most. This paper complicates that story. AI helps non-experts when it critiques. AI helps non-experts less when it leads. And AI actively gets in the way of experts when it leads. The tool’s value depends entirely on who’s using it and in what role.
What I find useful is the framework this gives for thinking about AI deployment in creative and knowledge work. The question isn’t “should we use AI for this task?” — it’s “what is the right role for AI in this task, given who’s doing it?” A company that rolls out an AI ghostwriting tool for its marketing team and assumes universal benefit may be helping junior copywriters while degrading the work of its most skilled people.
The sounding board modality is interesting precisely because it preserves human creative ownership while adding external perspective. This may be a more generalisable approach than ghostwriting — it helps non-experts without the anchoring risk that hurts experts. If you’re designing AI-assisted workflows for creative work, the default should probably be critique, not generation.