Assessing the predictive validity of strength of evidence grades a meta-epidemiological study

Depending on the definition used, C-indices ranged between 0.56 (95% CI, 0.47 to 0.66) and 0.58 (95% CI, 0.50 to 0.67) indicating a low discriminatory ability. CONCLUSIONS: The limited predictive validity of the EPC approach to GRADE seems to reflect a mismatch between expected and observed changes...

Full description

Main Author: Gartlehner, Gerald
Corporate Authors: United States Agency for Healthcare Research and Quality, RTI International-University of North Carolina Evidence-based Practice Center
Format: eBook
Language:English
Published: Rockville (MD) Agency for Healthcare Research and Quality (US) September 2015, 2015
Series:Research white paper
Online Access:
Collection: National Center for Biotechnology Information - Collection details see MPG.ReNa
LEADER 03619nmm a2200289 u 4500
001 EB001839818
003 EBX01000000000000001003807
005 00000000000000.0
007 cr|||||||||||||||||||||
008 180702 ||| eng
100 1 |a Gartlehner, Gerald 
245 0 0 |a Assessing the predictive validity of strength of evidence grades  |h Elektronische Ressource  |b a meta-epidemiological study  |c investigators, Gerald Gartlehner, Andreea Dobrescu, Tammeka Swinson Evans, Carla Bann, Karen A. Robinson, James Reston, Kylie Thaler, Andrea Skelly, Anna Glechner, Kimberly Peterson, Christina Kien, Kathleen N. Lohr 
260 |a Rockville (MD)  |b Agency for Healthcare Research and Quality (US)  |c September 2015, 2015 
300 |a 1 online resource (1 PDF file (vi, 18, A-23, pages))  |b illustrations 
505 0 |a Includes bibliographical references 
710 2 |a United States  |b Agency for Healthcare Research and Quality 
710 2 |a RTI International-University of North Carolina Evidence-based Practice Center 
041 0 7 |a eng  |2 ISO 639-2 
989 |b NCBI  |a National Center for Biotechnology Information 
490 0 |a Research white paper 
500 |a Title from PDF title page 
856 |u http://www.ncbi.nlm.nih.gov/books/NBK321518  |3 Volltext 
082 0 |a 610 
520 |a Depending on the definition used, C-indices ranged between 0.56 (95% CI, 0.47 to 0.66) and 0.58 (95% CI, 0.50 to 0.67) indicating a low discriminatory ability. CONCLUSIONS: The limited predictive validity of the EPC approach to GRADE seems to reflect a mismatch between expected and observed changes in treatment effects as bodies of evidence advance from insufficient to high SOE. In addition, many low or insufficient grades appear to be too strict 
520 |a For each grade of SOE, we compared the observed proportion of stable estimates with the expected proportion from an international survey. To determine the predictive validity, we used the Hosmer-Lemeshow test to assess calibration and the C (concordance) index to assess discrimination. RESULTS: Overall, the predictive validity of the EPC approach to GRADE for the stability of effect estimates was limited. Except for moderate SOE, the expected and observed proportions of stable effect estimates differed considerably. Estimates graded as high SOE were less likely to remain stable than expected by producers and users of systematic reviews. By contrast, estimates graded as low or insufficient SOE were substantially more likely to remain stable than expected. In this sample, the EPC approach to GRADE could not reliably predict the likelihood that individual bodies of evidence remain stable as new evidence becomes available.  
520 |a OBJECTIVE: We sought to determine the predictive validity of the U.S. Evidence-based Practice Center (EPC) approach to GRADE (Grading of Recommendations Assessment, Development and Evaluation) by examining how reliably it can predict the likelihood that treatment effects remain stable as new studies emerge. STUDY DESIGN AND SETTING: Based on 37 Cochrane reports with outcomes graded as high strength of evidence (SOE), we prepared 160 documents using portions of these bodies of evidence in a chronological order. We randomly assigned these documents, which represented different levels of SOE, to professional systematic reviewers from seven academic centers in Austria, Canada, and the United States, who dually graded the SOE using guidance for the EPC program. For each of the 160 documents, we determined whether estimates remained stable as subsequent studies were added to the evidence base.