LAWRENCE, Kan. — AI chatbot ChatGPT has dominated headlines since its release. The chatbot is undeniably impressive, capable of producing human-like text and conversations in response to written prompts, but researchers from the University of Kansas report that AI-produced content is still distinguishable from human writing – for the moment at least. In their new study, researchers explain there are many telltale signs that can help people distinguish AI chatbots from humans.
Using these signs, the research team at UK was able to develop a new tool for identifying AI-generated academic science writing, featuring over 99 percent accuracy.
“We tried hard to create an accessible method so that with little guidance, even high school students could build an AI detector for different types of writing,” says first study author Heather Desaire, a professor at the University of Kansas, in a media release. “There is a need to address AI writing, and people don’t need a computer science degree to contribute to this field.”
“Right now, there are some pretty glaring problems with AI writing,” she continues. “One of the biggest problems is that it assembles text from many sources and there isn’t any kind of accuracy check — it’s kind of like the game Two Truths and a Lie.”
While there are already a number of AI text detectors available online, and most perform fairly well, all of those models weren’t built specifically for academic writing. This time around, researchers set out to build a tool with better performance precisely for this purpose. To that end, they focused on a variety of articles called perspectives, which are used to provide an overview of specific research topics written by scientists.
The team selected a total of 64 perspectives, and then went on to create 128 ChatGPT-generated articles on the same research topics to train the model. When researchers compared those articles, they spotted signs of predictability — considered an indicator of AI writing.
Unlike AI, humans use more complex paragraph structures that vary in sentence amount and total words per paragraph, not to mention fluctuating sentence length as well. Preferences and quirks in punctuation marks and vocabulary are also a giveaway of human writing. For instance, scientists gravitate towards words like “however,” “but,” and “although,” while ChatGPT often uses “others” and “researchers” in writing. All in all, study authors collected 20 characteristics for the model to search for.
Upon testing, the model earned an astounding 100 percent accuracy rate at detecting AI-generated full perspective articles from those written by humans. Meanwhile, when it comes to identifying individual paragraphs within the article, the model earned an accuracy rate of 92 percent. The researchers’ model also outperformed an available AI text detector on the market by a rather wide margin during a round of similar tests.
Moving forward, researchers plan on determining the scope of the model’s applicability. Such assessments would entail testing it on more extensive datasets and across different types of academic science writing. Of course, AI chatbots are only going to continue advancing and improving, and researchers also want to know if their model will stand up against newer technologies.
“The first thing people want to know when they hear about the research is ‘Can I use this to tell if my students actually wrote their paper?’” Prof. Desaire concludes.
While this model appears to be highly skilled at both distinguishing between AI and scientists, Prof. Desaire stresses it was not designed to catch AI-generated student essays for educators. However, she adds that people can easily replicate their methods to build models serving their own purposes.
The study is published in Cell Reports Physical Science.
You might also be interested in:
- Confessions of ChatGPT: ‘Interview’ with bot reveals bizarre nature of questions — but I may have been fooled
- ‘An oppressive society’: Watch this humanoid robot reveal ‘nightmare scenario’ about AI taking over world
- ‘I used ChatGPT to force my landlord to fix my washer and dryer’
