Source: Bai, Boyson, Cao, Liu & Wan (2025) — “Executives vs Chatbots: Unmasking Insights Through Human-AI Differences in Earnings Conference Q&A”
At first this sounds like a technical finance question. Then you look closer and realise it is also a question about attention, incentives, and whether people are using information or just being impressed by it.
Earnings calls are one of the most scrutinised moments in a public company’s calendar. The whole ritual is predicated on the idea that something meaningful is being communicated — that the answers reveal something the market didn’t already know. But how do you actually measure whether that’s true? This paper introduces a clever way of thinking about the problem. The researchers use large language models — specifically ChatGPT and Google Bard — to generate what an AI would say in response to each analyst question, given the same context available to the market. They then compare those AI-generated answers to what the executive actually said. The gap between the two — they call it HAID, for Human-AI Difference — becomes a measure of genuine information content. The intuition is worth sitting with for a moment. If an executive’s answer sounds exactly like what an AI trained on public data would produce, there’s probably nothing new there. The executive is just restating what the market already knows in different words. But if the answer diverges substantially from the AI’s prediction, that divergence likely reflects something the AI couldn’t have known — private information, nuance, judgment, or disclosure that isn’t already priced in.
So I would not read this as a neat technology story. It is a messy information story. The tools improve, but people still have to decide what deserves attention. Annoying, but true.