ChatGPT, an artificial intelligence chatbot created by American company OpenAI, has exhibited the ability to rival the top 1% of human thinkers in a standard test of creativity, said research conducted by the University of Montana.
The AI application showcased exceptional fluency and originality in the Torrance Tests of Creative Thinking (TTCT), a widely recognised tool for assessing creative potential. Dr. Erik Guzik, an assistant clinical professor at UM's College of Business who directed the study, remarked, "For ChatGPT and GPT-4, we showed for the first time that it performs in the top 1% for originality. That was new."
To conduct the study, responses from 24 University of Montana students were collected and compared with scores from 2,700 students nationwide who also participated in the TTCT. Scholastic Testing Service assessed all submissions, unaware of the AI's involvement. The scores were then juxtaposed with those of college students who took the TTCT in 2016.
The research revealed that ChatGPT's responses demonstrated a level of creativity on par with the most imaginative human participants who took the test, and in fact, surpassed the creative abilities of a majority of students nationally. The researchers cautiously awaited the results and avoided hasty conclusions. However, the evidence strongly suggested that AI may be developing creative capabilities on a level equal to or even surpassing human abilities.
The Torrance Tests of Creative Thinking comprises two different assessments: one verbal and one figural, both measuring divergent thinking—the thought process used to generate innovative ideas.
ChatGPT's responses demonstrated remarkable fluency and originality, impressing the researchers with its novel and surprising outputs. Dr. Guzik remarked, "We had all been exploring with ChatGPT, and we noticed it had been doing some interesting things that we didn't expect. Some of the responses were novel and surprising. That's when we decided to put it to the test to see how creative it really is."
The study's findings open up exciting possibilities for the future development and application of AI technologies in creative endeavors and problem-solving tasks.
However, this comes a few days after another study pointed out an alarming decline in the quality of responses given by ChatGPT.
Researchers from Stanford University and UC Berkeley collaborated to investigate whether there was a decline in the quality of ChatGPT's performance and developed metrics to quantify the extent of any detrimental changes. The results of their study revealed that the deterioration in ChatGPT's quality was indeed a real phenomenon.
The research paper titled "How Is ChatGPT's Behavior Changing Over Time?" was authored by three distinguished academics: Matei Zaharia, Lingjiao Chen, and James Zou. The paper, which was recently published, delves into the evolving behavior of ChatGPT over a specific time period. Matei Zaharia, a Computer Science Professor at UC Berkeley, shared the eye-opening findings on Twitter.
Among the notable findings was a sharp decline in GPT-4's success rate in handling a specific task. The success rate on the task of determining whether a given number is prime and performing step-by-step thinking saw a significant drop from 97.6% to a mere 2.4% between the months of March and June 2023.
The research underscored the need for ongoing monitoring and evaluation of AI models to identify any changes in their behavior and performance. Understanding how AI systems evolve over time is crucial to ensuring their continued accuracy and reliability, especially when they are relied upon for critical tasks. The findings from this study will likely prompt further investigations and refinements in AI development to mitigate potential issues and maintain high-quality performance.