The AI-Mayo PE (AIM PE) Study: Validation of an Artificial Intelligence Algorithm Using Electrocardiograms to Predict Pulmonary Emboli

Background: Risk stratification for suspected pulmonary emboli (PE) remains an area of active research. Current risk stratification algorithms have moderate discriminatory capabilities and are guideline-recommended, but they are poorly utilized in practice, with most clinicians relying predominantly on their clinical training and personal “gestalt.” Our group recently derived an Artificial Intelligence (AI) algorithm using electrocardiograms (ECG) to predict the presence of PE (AUC 0.69) on computed tomography scans of the pulmonary arteries (CTPA) in a cohort of 79K patients.

Aim: Validate the AI-ECG algorithm in a contemporary dataset of patients with suspected PE.

Methods: Emergency Department (ED) visits between 9/1/2022 and 12/31/2023 across the Mayo Clinic Enterprise were searched using a unified data platform. ED visits in patients over 18 years of age with a discretely documented Wells’ PE or PERC Rule, a D-dimer result, or a CTPA were included. Only CTPA imaging and D-dimers performed within 48 hours of ED admission were eligible to be included. Patients were included in the final cohort if an ECG was performed within 6 hours of ED admission, D-dimer testing, or CTPA. Patients were considered negative for PE if CTPA was negative, and when this wasn’t performed, a negative D-dimer (high sensitivity) or PERC rule of 0 was considered negative. A previously derived Artificial Neural Network algorithm to analyze the ECG to predict PE was applied to the dataset for validation.

Results: A total of 27,028 ED visits in 24,288 unique patients meet inclusion criteria. Among this group, the first ED visit with a CTPA and an ECG within 6 hours was included, leaving 18,936 patients for the final validation cohort. The mean age of the cohort was 54.7 (SD 18.7), and 57% were male. Among the entire cohort, 13,573 underwent D-dimer testing for which 4,895 (36%) were positive at standard thresholds. A total of 10,486 patients underwent CTPA (55% of patients evaluated), of which 455 were positive for PE (4.3%). Using the AI-ECG model from the derivation cohort applied to this validation cohort resulted in an AUC of 0.69, identical to the derivation cohort. Using ECG model estimates alone, low (0.74%, n=2447, NPV 99.3%), elevated (2.27%, n=15404, NPV, 97.72%), and high-risk (8.02%, n=1085, NPV 91.98%) categories for PE can be created. A negative D-dimer in the elevated risk group increased the negative predictive value (NPV) to >99.9% and could be used to reduce imaging in this group further. Using the proposed ECG to D-dimer cascade risk stratification could further reduce the need for PE imaging for an overall NPV of 99.9% and sensitivity of 95.2%.

We then used additional clinical parameters to train a new algorithm with additional clinical data and the AI-ECG algorithm estimates. Multiple machine learning algorithms were then evaluated in a dataset divided into training (2/3) and testing (1/3) datasets, after downsampling (to address class imbalance). The best results were obtained with AI-ECG results and D-dimer: Ada Boost (AUC 0.93, F1 score 0.895) compared to standard logistic regression (AUC 0.79, F1 score 0.746). The addition of age, sex, and oxygen saturation did not improve model performance.

Conclusion: We have now validated the previously derived AI-ANN ECG algorithm for PE in a contemporary and independent cohort and demonstrate potential for integration into clinical PE assessment algorithms. Unlike prior risk stratification processes, ECGs are invariably performed in patients with cardiopulmonary symptoms and have the potential to prevent missed or delayed PE diagnosis by proactively identifying high-risk patients. Additionally, we demonstrate that the results of this algorithm can be combined with D-dimer values, resulting in a simple, powerful, predictive algorithm (AUC 0.93) that pairs cardiovascular electrophysiology with laboratory biomarkers and could lead to a new paradigm in clinical risk stratification for PE.