Comparison of effects as evidence evolves from single trials to high-quality bodies of evidence

OBJECTIVE: The objective of our methods project was to use a diverse sample of medical interventions to assess empirically whether first trials rendered substantially different treatment effect estimates than reliable, high-quality bodies of evidence. STUDY DESIGN AND SETTING: We employed a meta-epi...

Full description

Bibliographic Details
Main Author: Gartlehner, Gerald
Corporate Authors: United States Agency for Healthcare Research and Quality, RTI International-University of North Carolina Evidence-based Practice Center
Format: eBook
Language:English
Published: Rockville (MD) Agency for Healthcare Research and Quality (US) March 2015, 2015
Series:Research white paper
Subjects:
Online Access:
Collection: National Center for Biotechnology Information - Collection details see MPG.ReNa
LEADER 03211nam a2200277 u 4500
001 EB001839613
003 EBX01000000000000001003602
005 00000000000000.0
007 tu|||||||||||||||||||||
008 180702 r ||| eng
100 1 |a Gartlehner, Gerald 
245 0 0 |a Comparison of effects as evidence evolves from single trials to high-quality bodies of evidence  |h Elektronische Ressource  |c investigators, Gerald Gartlehner, Andreea Dobrescu, Tammeka Swinson Evans, Kylie Thaler, Barbara Nussbaumer, Isolde Sommer, Kathleen N. Lohr 
260 |a Rockville (MD)  |b Agency for Healthcare Research and Quality (US)  |c March 2015, 2015 
300 |a 1 PDF file (vi, 25 pages)  |b illustrations 
505 0 |a Includes bibliographical references 
653 |a Comparative Effectiveness Research 
653 |a Clinical Trials as Topic 
710 2 |a United States  |b Agency for Healthcare Research and Quality 
710 2 |a RTI International-University of North Carolina Evidence-based Practice Center 
041 0 7 |a eng  |2 ISO 639-2 
989 |b NCBI  |a National Center for Biotechnology Information 
490 0 |a Research white paper 
500 |a Title from PDF title page 
856 4 0 |u https://www.ncbi.nlm.nih.gov/books/NBK286188  |3 Volltext  |n NLM Bookshelf Books  |3 Volltext 
082 0 |a 610 
520 |a OBJECTIVE: The objective of our methods project was to use a diverse sample of medical interventions to assess empirically whether first trials rendered substantially different treatment effect estimates than reliable, high-quality bodies of evidence. STUDY DESIGN AND SETTING: We employed a meta-epidemiological study design using 100 bodies of evidence from Cochrane reports that had been graded as high quality of evidence. To determine the concordance of effect estimates between first and subsequent trials, we applied both quantitative and qualitative approaches. For quantitative assessment, we used Lin's concordance correlation and calculated z-scores; to determine the magnitude of differences of treatment effects, we calculated standardized mean differences (SMDs) and ratios of relative risks. We determined qualitative concordance based on a 2-tiered approach incorporating changes in statistical significance and magnitude of effect. RESULTS: First trials both over- and under-estimated the true treatment effects in no discernible pattern. Nevertheless, depending on the definition of concordance, effect estimates of first trials were concordant with pooled subsequent studies in at least 33 percent but up to 50 percent of comparisons. The pooled magnitude of change as bodies of evidence advanced from single trials to high-quality bodies of evidence was 0.16 SMD (95% confidence interval [CI], 0.12 to 0.21). In 80 percent of comparisons the difference in effect estimates was smaller than 0.5 SMDs. In first trials with large treatment effects (>0.5 SMD), however, estimates of effect substantially changed as new evidence accrued (mean change 0.68 SMD, 95% CI, .50 to 0.86) CONCLUSION: Results of first trials often change but the magnitude of change, on average, is small. Exceptions are first trials that present large treatment effects which often dissipate as new evidence accrues