PURPOSE

The aim of this study was to enhance multispecialty CT and MRI protocol assignment quality and efficiency through development, testing, and proposed workflow design of a natural language processing (NLP)-based machine learning classifier.

METHODS

NLP-based machine learning classification models were developed using order entry input data and radiologist-assigned protocols from more than 18,000 unique CT and MRI examinations obtained during routine clinical use. k-Nearest neighbor, random forest, and deep neural network classification models were evaluated at baseline and after applying class frequency and confidence thresholding techniques. To simulate performance in real-world deployment, the model was evaluated in two operating modes in combination: automation (automated assignment of the top result) and clinical decision support (CDS; top-three protocol suggestion for clinical review). Finally, model-radiologist discordance was subjectively reviewed to guide explainability and safe use.

RESULTS

Baseline protocol assignment performance achieved weighted precision of 0.757 to 0.824. Simulating real-world deployment using combined thresholding techniques, the optimized deep neural network model assigned 69% of protocols in automation mode with 95% accuracy. In the remaining 31% of cases, the model achieved 92% accuracy in CDS mode. Analysis of discordance with subspecialty radiologist labels revealed both more and less appropriate model predictions.

CONCLUSIONS

A multiclass NLP-based classification algorithm was designed to drive local operational improvement in CT and MR radiology protocol assignment at subspecialist quality. The results demonstrate a simulated workflow deployment enabling automated assignment of protocols in nearly 7 of 10 cases with very few errors combined with top-three CDS for remaining cases supporting a high-quality, efficient radiology workflow.