You probably think you prefer writing created by a human. But a new study is proving that you may not actually. You may just like things that sound human.
Led by Cambridge University Press and researchers at Villanova University, the study asked its 1,682 participants aged 18 to 81 to read a few fictional stories to determine whether they could tell if they were AI-generated, and then asked them to rate the quality of each story.
The study consisted of three separate experiments. In part one, all participants were given a story – some told it was AI-generated, others told it was human-written – and then rated its quality and engagement. In the second and third experiments, the participants were divided into two groups: one with 424 participants and the other with 481. Each person was given one human-written and one ChatGPT-generated story, though it was not disclosed which was which, and they were asked to identify which story they believed was authored by ChatGPT.
They found that the participants rated stories generated by ChatGPT highest, especially when they were intentionally and incorrectly told the story had been written by a human. Researchers also found that many (over 60% in experiment two and over 48% in experiment three) couldn’t correctly identify which content was AI-generated.
Deena Weisberg, senior author of the study and professor in the Department of Psychological & Brain Sciences at Villanova University, said this shows a bias toward “narratives written by real people.”
“We assume creative writing requires uniquely human qualities, such as emotional understanding and lived experience, which leads people to underestimate AI’s capabilities,” she said. “I’d say that public assumptions about AI’s capabilities are increasingly out of date.”
Why do people actually prefer AI writing?
There is a growing portfolio of evidence that people like AI-generated writing, including short stories, essays and poetry.
A 2023 study led by Renee Richardson Gosline of MIT found similar results, concluding that AI-generated content and augmented AI (the AI made the final decision with human output as a reference) were seen as higher quality than that produced by human experts and augmented human experts (the human makes the final decision with AI output as a reference).
Weisberg speculates that this is likely due to AI-written stories being more direct, explicit and simple, whereas human-written stories tend to be more subtle or complex and leave room for interpretation.
For instance, a snippet from one of the AI-generated stories given to the participants reads:
“So now, as I sit by the pond, the autumn leaves falling gently around me, I think of my mother, of the stories she shared, of the wisdom she imparted. The koi continue their dance beneath the water’s surface, oblivious to the change they have witnessed in my life. And in their simple, unchanging world, I find a sense of comfort, a reminder that life, in all its complexity and uncertainty, goes on.”
Comparatively, a human-generated story has a portion that reads:
“Perhaps, I think, we are all experiencing a kind of birth after all, she laboring and I midwifing, her husband playing the role of nervous father, not sure what he’ll do after the sacred event occurs, not sure what to do now. We swim in uneasy circles around each other, and all the while the golden sun continues to rise and set, keeping its own mysterious time.”
The AI-generated text as a whole is much more direct, uses simpler sentences, doesn’t rely on anecdotes or literary devices like similes and metaphors, and has much shorter paragraphs.
In essence, it’s not that the participants have bad taste or that the large language model powering the chatbot has any skill a human writer lacks; it’s more that AI writing often incorporates tropes, syntax, language and styles that people already like. This outcome may not be all that surprising when we’re reminded that these AI systems are trained on writing we, as humans, have enjoyed and consumed for decades.
“We hear a lot about the crisis in literacy, shrinking attention spans and the influence of platforms, such as TikTok,” Weisberg said. “But I don’t think we can say that our findings in this study can be entirely explained by contemporary media consumption.”
“It’s more that these technologies amplify existing tendencies, rather than creating them.”
See full bio