Robot consulting stock market investment financial dashboard

(© tiagozr - stock.adobe.com)

LAWRENCE, Kan. — AI chatbot ChatGPT has dominated headlines since its release. The chatbot is undeniably impressive, capable of producing human-like text and conversations in response to written prompts, but researchers from the University of Kansas report that AI-produced content is still distinguishable from human writing – for the moment at least. In their new study, researchers explain there are many telltale signs that can help people distinguish AI chatbots from humans.

Using these signs, the research team at UK was able to develop a new tool for identifying AI-generated academic science writing, featuring over 99 percent accuracy.

“We tried hard to create an accessible method so that with little guidance, even high school students could build an AI detector for different types of writing,” says first study author Heather Desaire, a professor at the University of Kansas, in a media release. “There is a need to address AI writing, and people don’t need a computer science degree to contribute to this field.”

“Right now, there are some pretty glaring problems with AI writing,” she continues. “One of the biggest problems is that it assembles text from many sources and there isn’t any kind of accuracy check — it’s kind of like the game Two Truths and a Lie.”

While there are already a number of AI text detectors available online, and most perform fairly well, all of those models weren’t built specifically for academic writing. This time around, researchers set out to build a tool with better performance precisely for this purpose. To that end, they focused on a variety of articles called perspectives, which are used to provide an overview of specific research topics written by scientists.

The team selected a total of 64 perspectives, and then went on to create 128 ChatGPT-generated articles on the same research topics to train the model. When researchers compared those articles, they spotted signs of predictability — considered an indicator of AI writing.

Unlike AI, humans use more complex paragraph structures that vary in sentence amount and total words per paragraph, not to mention fluctuating sentence length as well. Preferences and quirks in punctuation marks and vocabulary are also a giveaway of human writing. For instance, scientists gravitate towards words like “however,” “but,” and “although,” while ChatGPT often uses “others” and “researchers” in writing. All in all, study authors collected 20 characteristics for the model to search for.

ChatGPT prompt
(Photo by Om siva Prakash on Unsplash)

Upon testing, the model earned an astounding 100 percent accuracy rate at detecting AI-generated full perspective articles from those written by humans. Meanwhile, when it comes to identifying individual paragraphs within the article, the model earned an accuracy rate of 92 percent. The researchers’ model also outperformed an available AI text detector on the market by a rather wide margin during a round of similar tests.

Moving forward, researchers plan on determining the scope of the model’s applicability. Such assessments would entail testing it on more extensive datasets and across different types of academic science writing. Of course, AI chatbots are only going to continue advancing and improving, and researchers also want to know if their model will stand up against newer technologies.

“The first thing people want to know when they hear about the research is ‘Can I use this to tell if my students actually wrote their paper?’” Prof. Desaire concludes.

While this model appears to be highly skilled at both distinguishing between AI and scientists, Prof. Desaire stresses it was not designed to catch AI-generated student essays for educators. However, she adds that people can easily replicate their methods to build models serving their own purposes.

The study is published in Cell Reports Physical Science.

You might also be interested in:

About John Anderer

Born blue in the face, John has been writing professionally for over a decade and covering the latest scientific research for StudyFinds since 2019. His work has been featured by Business Insider, Eat This Not That!, MSN, Ladders, and Yahoo!

Studies and abstracts can be confusing and awkwardly worded. He prides himself on making such content easy to read, understand, and apply to one’s everyday life.

Our Editorial Process

StudyFinds publishes digestible, agenda-free, transparent research summaries that are intended to inform the reader as well as stir civil, educated debate. We do not agree nor disagree with any of the studies we post, rather, we encourage our readers to debate the veracity of the findings themselves. All articles published on StudyFinds are vetted by our editors prior to publication and include links back to the source or corresponding journal article, if possible.

Our Editorial Team

Steve Fink

Editor-in-Chief

Chris Melore

Editor

Sophia Naughton

Associate Editor