Development and Preliminary Usability and Accuracy Testing of the EBMT Gvhd App to Support Graft Versus Host Disease Diagnosis and Scoring According to NIH Criteria, By the EBMT Complications and Quality of Life Working Party

Introduction: Accurate diagnosis and severity scoring of acute and, in particular, chronic GVHD remains a challenge for clinical practice and for correct self-reporting of GVHD data to evaluate transplant outcomes. Perceived complexity and time investment issues limit the implementation of international standards and the recently updated NIH criteria for chronic GVHD. Here, we describe the development of the EBMT GVHD App, a computer/web-based algorithm-driven application to help physicians correctly diagnose and score severity of acute and chronic GVHD and improve the implementation of current standards in clinical practice and research.

Methods: We applied a User Centered Design (UCD) process, through an iterative process between end-users and the development team to ensure that the App is user-friendly and efficient. A first EBMT GVHD App version (v0.0) tested an initial GVHD algorithm. An second improved prototype v1.0 was developed as a true web application (App) compatible with desktop computers and smartphones/tablets. V1.0 relies on two modules: a diagnostic module for NIH diagnostic, distinctive, common signs or proven evidence of GVHD (skin, nails, scalp/body hair, mouth, eyes, genitals, gastrointestinal tract, liver, lungs and muscles/joints), and a scoring module to assess severity of acute (Glucksberg and IBMTR criteria), chronic and overlap (NIH criteria) GVHD. The App v1.0 was tested by 28 hematology professionals (University Hospitals of Leuven, Belgium): 8 senior physicians, 8 junior physicians, 2 medical students and 10 data managers/research nurses; median experience in hematology 2.25 years, range 0-30, IQR 6.6, evenly distributed, by profession and seniority, to one of two groups (A and B).

Usability of the App was determined for user experience and satisfaction. User experience was tested at baseline and end of study with the technology acceptance model (TAM), evaluating six Perceived Usefulness Statements rated on a 7-point Likert-like scale (1= extremely unlikely to 7= extremely likely). User satisfaction was evaluated by PSSUQ (Post-Study System Usability Questionnaire), which scores system usefulness, information quality and interface quality, on a 7-point Likert-like scale (1= strongly agree to 7= strongly disagree). App’s accuracy relied on the proportion of correctly assessed clinical scenarios from 4 representative GVHD cases developed by a panel of GVHD experts as gold standard. In a quasi-experimental crossover design, professionals were invited to solve two cases either with standard paper tools or with the App, and later crossed over to use the other tool, both for the two other cases, as well as to solve again with the new tool their previous two cases. Comparisons were performed ‘within groups’ and ‘between groups’ (A vs B).

Results: User experience and satisfaction were very good, with a median of 6 TAM points for user experience, and a median overall PSSUQ score of 2.2 for user satisfaction, 2.1 for System Use, 2.4 for Information Quality and 1.7 for Interface Quality. Users (70%) reported that they would be more likely to use the App on a desktop than on a mobile device.

Accuracy results were only moderate with standard paper tools: 65% for diagnosis and 45% for scoring. The use of the App significantly increased diagnostic and scoring accuracy to 94% (+29%) and 88% (+43%), respectively (both p<0.001). The App also improved accuracy of individuals repeating the same clinical case (within groups) for diagnosis (+27%) and scoring (+42%), beyond a potential learning effect.

From v1.0 results, an App v2.0 has been developed refining details in the algorithm, improving term description, adding a user’s manual and the option of generating patient reports, which is now ready for further testing.

Conclusions: The “EBMT GVHD App” is a first electronic tool to diagnose and score GVHD. Initial testing of v1.0 uniformly showed high scores for user experience and satisfaction, accurately reflected the subtle nuances of the NIH criteria, and improved significantly the accuracy of a diverse group of hematology professionals to diagnose and score severity of GVHD, compared to their practice with standard tools. Testing of v2.0 is underway to adapt layout and screen content and to address ambiguities of current guidelines. A larger study with a subsequent v3.0 is warranted in real life setting to evaluate macroscopic scalability.