Large language models for extraction of OPS-codes from operative reports in meningioma surgery.

BACKGROUND

In the German medical billing system, surgical departments encode their procedures in OPS-codes. These OPS-codes have major impact on DRG grouping and thus mainly determine each case´s revenue. In our study, we investigate the ability of the Large Language Model (LLM) GPT to derive correct OPS codes from the surgical report.

METHODS

For our study, 100 patients who underwent meningioma surgery at our clinic between 2023 and 2024 were examined. We recorded the OPS codes assigned by the surgeon after the procedure, as well as the final coding by the hospital´s coders before case closure. In addition, the surgical report was extracted and anonymously provided to GPT-4o and GPT CodeMedic together with the current OPS-catalogue. The coding of each group was analyzed descriptively and compared using the Chi-Square test. Additionally, errors and deviations were assessed and analyzed.

RESULTS

In our analyses, coders (100%) and surgeons (99%) demonstrated to significantly perform higher than LLMs in sufficient coding, for which the basic coding must be correct and unquestionable (GPT-4o 78%, GPT CodeMedic 89%; p < 0.01). For optimal coding, where every code potentially contributing to increase the revenue must be included, only the coders (94%) achieved superiority (GPT-4o p < 0.01; GPT CodeMedic p = 0.02), whereas GPT CodeMedic (83%) even outperformed surgeons (69%) (p = 0.03). The specialized GPT CodeMedic tends to show fewer hallucinations compared to GPT-4o (7% vs. 15%).

CONCLUSION

GPT is capable of extracting OPS codes from surgical reports. The most frequent errors made by LLMs can be attributed to a lack of specialized training. Currently, professional coders still significantly outperform LLMs in sufficient and optimal coding. For optimal coding however, GPT shows to perform comparably to surgeon´s coding skills. This indicates, that in near future after further training, LLMs may take over this task from surgeons without loss in quality.