To determine whether natural language processing models can extract and/or predict the Expanded Disability Status Scale (EDSS) score from patients’ electronic health records. 

The EDSS is a widely used measure of monitoring disability progression in patients with multiple sclerosis (MS). However, extracting the EDSS from unstructured electronic health records can be labour intensive. Moreover, the EDSS may not be determined at all visits but can be derived from the contents of the consult note.

We studied 16,441 neurology consult notes for 4,808 patients who were followed at Canada’s largest MS outpatient clinic between June 2015 and July 2019. We randomly divided the records into training (80%) and test (20%) data sets. We compared the performance characteristics of three natural language processing models. First, we applied a keyword search approach, extracting the EDSS total score from sentences containing the keyword ‘EDSS’; next, we trained a machine learning (ML) model to predict the nineteen half-step increments of the EDSS score. Finally, we used a combined keyword-ML model.  For each approach, we determined the accuracy, positive predictive value (PPV), sensitivity and area under the Receiver Operating Characteristic Curve (ROC).

Overall, the combined keyword-ML model demonstrated the best performance, with accuracy, PPV, sensitivity and ROC of 0.89, 0.84, 0.83 and 0.86, respectively. Respective figures for the keyword and ML models individually were 0.62, 0.90, 0.69, 0.79 and 0.88, 0.74, 0.71, 0.93. Model performance for EDSS sub-scores (i.e. functional systems) was similar.

A combined keyword-ML natural language processing model can extract and predict EDSS scores and sub-scores from neurology consult notes. This approach can be automated for the extraction of clinically relevant information from unstructured notes, which enables the establishment of large databases that can facilitate clinical care and research for people living with MS.