Machine Learning Approach to Estimating ECOG PS for a Multiple-Myeloma Cohort from Real World Data

Background: ECOG Performance Status Scale (ECOG PS) is a measure of patient’s ability to function, and an important clinical attribute used in treatment decisions and trial recruitment. In the real world, therapies might be used for patients with more advanced disease, lower functional status, and a higher dependency on others for daily activities, and this population is often excluded from registry trials. This is particularly important for multiple myeloma (MM) in which patients often present with advanced disease and require new therapies despite their high ECOG scores. Of note, in real-world data sources (RWD), ECOG PS is often not recorded by physicians in either structured or unstructured clinical notes. To better understand patient and physician preferences amongst the new treatments available and to generate insights into distinct clinical subgroups, we developed a machine learning (ML) method to estimate ECOG PS for a multiple myeloma cohort using a curated RWD data set.

Methods: A curated RWD data set of 126K patients diagnosed with MM was derived from a federated Electronic Health Record network of 57 Healthcare Organizations in the US. Natural Language Processing (NLP) methods were used to extract ECOG scores from clinical notes for 4,799 patients and used as ground truth for model building. Based on input from clinical experts, we identified 96 features including MM specific features such as disease characteristics including, ISS Staging, CRAB values, etc. as well as features indicative of patient performance such as use of orthotic device, stroke/brain injury, days in inpatient settings, etc. Our initial experiments suggested that non-MM specific variables may contribute significantly to the performance of the model. For our final model building we included 38 non-MM specific features after subtracting features of low variance and collinearity. Dataset was divided into training, testing and validation sets with a split of 60, 20, 20 respectively. We used an XGBoost algorithm and tuned hyper-parameters for the classifier. We tested both multiclass (ECOG PS 0-4) and binary (0-1, 2-4) classifiers. The resulting model was used to estimate ECOG scores longitudinally, first at time of diagnosis and subsequently on a date which meets all of the following: at least one year after diagnosis, had key features available and prior to discontinuation of MM treatments. Succeeding ECOG scores were estimated at least 365 days apart to ensure patients were monitored along their treatment.

Results: For multiclass classification, our model achieved ROC-AUC (receiver operating characteristic area under the curve) of 0.91, F1 scores of 0.76, precision and recall scores of 0.87 and 0.70 respectively. For binary classification F1 was 0.86 and precision and recall scores were 0.96 and 0.8 respectively. Body mass index (BMI), age, the number of outpatient visits and total amount of days spent in an inpatient setting were found to be the most important features in predicting ECOG PS, with next-most important being co-morbid diagnoses of a stroke/brain injury and difficulty walking. Using the model, we estimated ECOG scores for an additional 30K patients resulting in a total of 58K longitudinal ECOG scores. Median time between estimation dates was 394 days to match how MM patients were monitored prior to discontinuation of any MM treatments.

Conclusions: ECOG PS can have wide variance including inter-operator discordance (19%-56%, Datta SS et al. Ecancermedicalscience, 2019; 13:913) and discordance between physician and patient reported scores (57%, Schnadig ID et al. Cancer. 2008). In this context, our multiclass and binary classifiers achieved positive predictive values 0.87 and 0.96 respectively using non-MM clinical features. Given the real-world variability in ECOG estimation between physicians and between patients and physicians, performance characteristics of our model are appropriate and acceptable in deriving clinical insights using estimated ECOG PS. To further validate our model, work to characterize and compare outcomes in our cohort using NLP and ML derived ECOG scores will be carried out. In addition, since our model uses non-oncology features to estimate ECOG PS, we will continue to test validity in additional liquid and solid tumors.