Usage of AI in research papers deteriorates quality: Analysis

New Delhi: The use of artificial intelligence in writing research papers is reshaping scientific publishing with an increase in the use of complex language and a decline in research quality, according to an analysis of over 2.1 million preprints and peer-reviewed studies, PTI reported.

Findings published in the journal Science come at a time when journals, including those run by Springer Nature and Elsevier, are explicitly allowing and publishing guidelines around the use of AI in supporting and writing research.

Researchers from the US's Cornell and California (Berkeley) universities said that advancing AI systems will "challenge our fundamental assumptions about research quality, scholarly communication, and the nature of intellectual labour".

"The scientific enterprise is intimately connected with technological innovation, and 'science policy-makers must consider how to evolve our scientific institutions to accommodate the rapidly changing scientific production process,' the authors said.

Amidst growing enthusiasm -- alongside concerns -- around the use of generative AI and large language models in research and academia, there is little systemic evidence on how the technologies are reshaping scientific production, the team said.

Five datasets spanning 2.1 million preprints, 28,000 peer-reviewed studies, and 246 million online views and downloads of scientific documents were analysed.

Text-based detectors were used to identify the first-time use of a large language model -- AI systems that can process natural language of humans -- which helped compare researchers' work before and after resorting to the use of AI.

Use of large language models could boost a scientist's research productivity by 23 per cent to nearly 89 per cent, with especially large boosts for those facing more writing and language barriers, the analysis found.

Scholars with Asian names, affiliated with institutions in Asia, were estimated to see a productivity gain between 43 per cent and 89.3 per cent linked with the use of AI.

Caucasian-named researchers, affiliated with institutions in English-speaking countries, saw a "modest, but significant" increase in scientific output of 23.7 per cent to over 46 per cent, the study found.

Adoption of large language models was also associated with a more sophisticated language, but in substantively weak manuscripts. Traditionally, a complex writing language has been related to a higher research quality.

The result "confirms that complex LLM-generated language often disguises weak scientific contributions, the authors said.

Use of large language models may also change an author's citation behaviour, steering them towards a more diverse knowledge base, including more books and younger, less-cited articles, the team found.

Nearly 12 per cent of the researchers analysed were found to cite more books, possibly reflecting an AI model's ability to surface content from lengthy texts, the analysis said.

"Our findings show that LLMs (large language models) have begun to reshape scientific production. These changes portend an evolving research landscape in which the value of English fluency will recede, but the importance of robust quality-assessment frameworks and deep methodological scrutiny is paramount," the authors wrote.

"For peer reviewers and journal editors, and the community, more broadly, who create, consume, and apply this work, this represents a major issue," they said.