Evaluating Artificial Intelligence-Based Writing Assistance Among Published Orthopaedic Studies: Detection and Trends for Future Interpretation.

BACKGROUND

The integration of artificial intelligence (AI), particularly large language models (LLMs), into scientific writing has led to questions about its ethics, prevalence, and impact in orthopaedic literature. While tools have been developed to detect AI-generated content, the interpretation of AI detection percentages and their clinical relevance remain unclear. The aim of this study was to quantify AI involvement in published orthopaedic manuscripts and to establish a statistical threshold for interpreting AI detection percentages.

METHODS

To establish a baseline, 300 manuscripts published in the year 2000 were analyzed for AI-generated content with use of ZeroGPT. This was followed by an analysis of 3,374 consecutive orthopaedic manuscripts published after the release of ChatGPT. A 95% confidence interval was calculated in order to set a threshold for significant AI involvement. Manuscripts with AI detection percentages above this threshold (32.875%) were considered to have significant AI involvement in their content generation.

RESULTS

Empirical analysis of the 300 pre-AI-era manuscripts revealed a mean AI detection percentage (and standard deviation [SD]) of 10.84% ± 11.02%. Among the 3,374 post-AI-era manuscripts analyzed, 16.7% exceeded the AI detection threshold of 32.875% (2 SDs above the baseline for the pre-AI era), indicating significant AI involvement. No significant difference was found between primary manuscripts and review studies (percentage with significant AI involvement, 16.4% and 18.2%, respectively; p = 0.40). Significant AI involvement varied significantly across journals, with rates ranging from 5.6% in The American Journal of Sports Medicine to 38.3% in The Journal of Bone & Joint Surgery (p < 0.001).

CONCLUSIONS

This study examined AI assistance in the writing of published orthopaedic manuscripts and provides the first evidence-based threshold for interpreting AI detection percentages. Our results revealed significant AI involvement in 16.7% of recently published orthopaedic literature. This finding highlights the importance of clear guidelines, ethical standards, responsible AI use, and improved detection tools to maintain the quality, authenticity, and integrity of orthopaedic research.