A Natural Language Processing Pipeline based on the Columbia-Suicide Severity Rating Scale.

IMPORTANCE

Diagnostic codes in the Electronic Health Record (EHR) are known to be limited in reporting patient suicidality, and especially in differentiating the levels of suicide severity.

OBJECTIVE

The authors developed and validated a portable natural language processing (NLP) algorithm for detection of suicidal ideation (SI) and suicide-related behavior and attempts (SB/SA) in EHR data. The algorithm was then deployed, and SI and SB/SA ascertainment was compared to that of International Statistical Classification of Diseases (ICD-9 and 10) diagnostic codes.

DESIGN

A group of experts designed the pipeline to detect and distinguish suicide severity based on the Columbia-Suicide Severity Rating Scale (C-SSRS). Notes were manually annotated to create the "Gold Standard" with which the algorithm output was evaluated for accuracy.

SETTING

The algorithm was developed at two academic medical centers, Weill Cornell Medicine (WCM), the Mount Sinai Health System (MSHS), and tested at these two, plus a third, the University of Utah Healthcare Center (UUHSC).

PARTICIPANTS

Notes were from participants with psychiatric encounters at the three institutions.

MAIN OUTCOMES

The two main outcomes were the accuracy scores of the NLP pipeline and comparison of ascertainment rates to ICD codes.

RESULTS

F1 accuracy scores ranged from 0.86-0.97 at the three sites. The NLP rate of detection of SB/SA was almost 30 times higher, and SI was almost 10 times higher, when compared with that of diagnostic codes. NLP detected almost all cases detected by diagnostic codes. No bias in performance was found for race/ethnicity and performance was comparable in psychiatric and non-psychiatric EHRs.

CONCLUSIONS AND RELEVANCE

EHRs from cohorts with psychiatric diagnoses or encounters at WCM, MSHS, and UUHSC had SI and SB/SA extracted using an NLP algorithm based on parameters defined by the C-SSRS. Validity was determined by comparing the algorithm output to manual annotations of clinical notes by domain experts. NLP-detection of SI and SB/SA was compared with that of ICD codes across a range of demographic groups. Algorithm performance was also examined for bias in minoritized groups and in non-psychiatric notes.

KEY POINTS

Can we automate the extraction of data available in clinical notes to accurately detect and distinguish patients with suicidal ideation (SI) and suicidal behavior (SB)? Our Natural Language Processing (NLP) approach was able to identify and distinguish SI and SB at three different hospital systems with benchmarked accuracy scores (above 0.85). The rate of detecting SI and SB using the algorithm was 10-30 times that of diagnostic codes found in the Electronic Health Record. Our algorithm renders the use of International Classification of Disease (ICD) diagnostic codes for SI and SB ascertainment obsolete.