-Author name in bold denotes the presenting author
-Asterisk * with author name denotes a Non-ASH member
Clinically Relevant Abstract denotes an abstract that is clinically relevant.

PhD Trainee denotes that this is a recommended PHD Trainee Session.

Ticketed Session denotes that this is a ticketed session.

4707 How Do ASH Guidelines Panels Make Decisions? Association between Decision Making Factors and the Strength of Recommendations

Program: Oral and Poster Abstracts
Session: 901. Health Services Research—Non-Malignant Conditions: Poster III
Hematology Disease Topics & Pathways:
Clinically relevant, Quality Improvement
Monday, December 3, 2018, 6:00 PM-8:00 PM
Hall GH (San Diego Convention Center)

Benjamin Djulbegovic, MD1, Tea Reljic, MPH2,3*, Shira Elqayam, PhD4*, Adam Cuker, MD, MS5, Iztok Hozo, PhD6*, Qi Zhou, PhD7*, Shelly-Anne Li, PhD(c)8*, Paul Alexander, PhD9*, Nieuwlaat Robby, PhD9*, Wojtek Wiercioch, MD9*, Holger Schünemann, MD10,11 and Gordon Guyatt, MD12*

1City of Hope, Duarte, CA
2University of South Florida, Tampa, FL
38491, Tampa, FL
4De Montfort University, Leicester,, United Kingdom
5Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA
6Indiana University, Gary
7Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Canada
8University of Toronto, Toronto, Canada
9McMaster University, Hamilton, Canada
10Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, ON, Canada
11Department of Medicine, McMaster University Health, Hamilton, Canada
12Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, ON, Canada

Background: Clinical practice guidelines (CPGs) represent a key mechanism for optimizing health care decision making. CPGs are a product of discussion by a group of, typically, 10-20 individuals with varying expertise. Little is known how CPG panels actually make their recommendations.

Objective: In a study of real-life decision making, we investigated the factors considered by members of panels convened by the American Society of Hematology (ASH) to develop guidelines, when using the widely accepted formal GRADE (Grading of Recommendations Assessment, Development, and Evaluation) system.

Methods: To account for panel level factors and individual level factors, we employed two level hierarchical, random-effect, multivariate logistic and ordered logistic regression analysis.

Results: 101 participants taking part in 8 CPGs panels issued 1,289 recommendations. Association of GRADE (normative) factors with the strength of recommendations (SOR) dominated the findings over the non-GRADE (descriptive) factors. In the main analysis certainty in evidence (regardless of direction for or against intervention) [OR=1.83 (95CI% 1.45 to 2.31;p<0.0001)], balance of benefits and harms [OR=1.49 (95CI% 1.30 to 1.69;p<0.0001)] and variability or uncertainty in the patients’ values and preferences [OR=1.47 (95CI% 1.15 to 1.88;p<0.002)] were the strongest predictors of SOR coded as “neither for nor against” , “weak for or against” or “strong for or against” health intervention. Greater judgment of certainty of evidence proved highly associated with a strong recommendation [OR=3.60 (95% CI 2.16 to 6.00;p<0.00001] when panel members were issuing recommendation “for” interventions. When, however, panels made recommendations “against” intervention, certainty in evidence was not associated with probability of issuing strong recommendation [OR=0.98 (95%CI: 0.57 to 1.8; p=0.94)]. Two panelist characteristics were associated with strong recommendations: age (per decade) [OR=1.79 (95 CI% 1.2 to 2.84; p<0.005)], and greater intolerance of uncertainties [OR=0.57 (95 CI% 0.37 to 0.86; p<0.008)]. Agreement between individual panel members and the group ranged from very poor (average kappa of -0.01 in one panel) to moderate (kappa=0.64 in another panel), with most panels in an intermediate range. We also found that the panel members who were asked by the ASH to recuse themselves from voting due to high risk of conflict of interest (COI) would have voted differently if they were allowed to do so.

Conclusion: Factors associated with GRADE’s conceptual framework proved, in general, highly associated with strong versus weak recommendations and with the direction of recommendation. However, some non-GRADE factors of importance for decision-making were also identified. Findings that panel members with high risk of COI made different judgments than those without COI provide empirical support for the importance of managing conflict of interest. The low agreement between individual panel members and group consensus, and failure of certainty of evidence to be associated with strength of recommendations against an intervention, suggest the need for improvements in the process.

Disclosures: Cuker: Synergy: Consultancy; Genzyme: Consultancy; Kedrion: Membership on an entity's Board of Directors or advisory committees; Spark Therapeutics: Research Funding.

*signifies non-member of ASH