Methods for evaluating natural experiments in obesity systematic evidence review
Most natural experiment studies were rated as having a "weak" global rating (i.e., high risk of bias), with 63 percent having a weak rating for handling of withdrawals and dropouts, 42 percent having a weak rating for study design, 40 percent having a weak rating for confounding, and 26 pe...
| Main Author: | |
|---|---|
| Corporate Authors: | , |
| Format: | eBook |
| Language: | English |
| Published: |
Rockville, MD
Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services
2017, December 2017
|
| Series: | Comparative effectiveness review
|
| Subjects: | |
| Online Access: | |
| Collection: | National Center for Biotechnology Information - Collection details see MPG.ReNa |
| Summary: | Most natural experiment studies were rated as having a "weak" global rating (i.e., high risk of bias), with 63 percent having a weak rating for handling of withdrawals and dropouts, 42 percent having a weak rating for study design, 40 percent having a weak rating for confounding, and 26 percent having a weak rating for data collection. Experimental studies were rated as "strong" (low risk of bias) in study design, control of confounding, and data collection methods, but were weaker in blinding and selection bias. We identified methodological and analytic advances that would help to strengthen efforts to estimate the effect of programs, policies, or built environment changes on obesity prevention and control, such as consistent use of data dictionaries, reporting standards on linkage methods of data sources, data sources with long-term public health surveillance of obesity and health behavioral outcomes, and use of study designs with multiple pre- and post-exposure time points. OUTCOMES/MEASURES: Of the 294 included studies, we identified 112 studies with childhood weight measures, 32 studies with adult weight measures, 152 studies with physical activity measures, and 148 studies with dietary measures. Thirty-seven of the 294 studies reported on outcomes related to the food environment, physical activity environment, commuting behavior, or purchasing behavior. STUDY DESIGN AND METHODS: Natural experiment studies most commonly used cross-sectional comparisons of exposed and unexposed groups (n=55; 35%). Difference-in-differences approaches that compared exposed and unexposed groups before and after an exposure were used in 45 studies (29%), while 48 studies (31%) used pre/post designs that compared one group before and after an exposure. Criteria for a data system (data source exists, is available for research, is sharable, and contains outcomes of interest) were met by 106 data sources (71 U.S. and 35 non-U.S.). Sixty-two percent of the U.S. data systems contained at least one of the main measures for weight or body mass index in adults or children, or dietary or physical activity behaviors. Fifty-three percent of the U.S. data systems included at least one outcome related to the food environment, physical activity environment, commuting behavior, or purchasing behavior, or included information about a relevant exposure in a policy, program, or built environment change. These 71 U.S. data systems often reported more than one outcome. Thirty-seven percent of the U.S. data systems were linked with a secondary data source or system other than the primary data source. Most studies that linked their data systems with external data systems used an individual-level key or a geographic allocation. CONCLUSIONS: Our systematic review identified numerous natural experiment studies (n=156) and data sources, including sharable and non-sharable data sources (n=216), that have been used to estimate the effect of programs, policies, or built environment changes on obesity prevention and control. The studies used a wide variety of outcome measures and analytic methods, often with substantial risk of bias. The findings reinforce the need for methodological and analytic advances that would strengthen efforts to improve obesity prevention and control OBJECTIVES: Obesity is an enormous public health problem among adults and children. Our objective was to systematically review studies evaluating programs and policies addressing obesity prevention and control in terms of their population-based data sources, use of data linkages, measures reported, study designs, and analytic approaches. The overarching goal of the review was to identify methodological advances that could strengthen research that uses natural experiments to evaluate the effectiveness of policies and programs to prevent and control obesity. DATA SOURCES: We systematically searched PubMed(r), CINAHL(r), PsycINFO(r), and EconLit from 2000 to August 21, 2017, to identify all U.S. and non-U.S. studies of programs or policies targeting obesity prevention and control in people of all ages and in any setting. REVIEW METHODS: Two independent reviewers screened abstracts and full-text articles. We required articles to be in English; address a program, policy, or built environment change; include 100 or more study subjects; and have a defined comparison or unexposed group. We used the Effective Public Health Practice Project (EPHPP) tool to rate studies for their risk of bias. This tool rates studies for their ability to draw causal inferences about program effectiveness. RESULTS: The search identified 26,316 unique citations. Of the 294 studies (reported in 312 articles) eligible for inclusion (188 U.S. and 106 non-U.S.), 156 (53%) were natural experiment studies, 118 (40%) were experimental studies (randomized or nonrandomized controlled trials), and 20 (7%) had other study designs that did not fall into either of the other categories.DATA SOURCES: We identified 143 secondary data sources and 26 sharable primary data sources, totaling 116 sharable data sources after duplicates were removed. |
|---|---|
| Physical Description: | 1 PDF file (various pagings) illustrations |