文献检索文档翻译深度研究
Suppr Zotero 插件Zotero 插件
邀请有礼套餐&价格历史记录

新学期,新优惠

限时优惠:9月1日-9月22日

30天高级会员仅需29元

1天体验卡首发特惠仅需5.99元

了解详情
不再提醒
插件&应用
Suppr Zotero 插件Zotero 插件浏览器插件Mac 客户端Windows 客户端微信小程序
高级版
套餐订阅购买积分包
AI 工具
文献检索文档翻译深度研究
关于我们
关于 Suppr公司介绍联系我们用户协议隐私条款
关注我们

Suppr 超能文献

核心技术专利:CN118964589B侵权必究
粤ICP备2023148730 号-1Suppr @ 2025

国际发展中的研究比较与偏倚风险:系统评价与批判性评估。

Within study comparisons and risk of bias in international development: Systematic review and critical appraisal.

作者信息

Villar Paul Fenton, Waddington Hugh

机构信息

School of International Development University of East Anglia Norwich UK.

International Initiative for Impact Evaluation (3ie) and London School of Hygiene and Tropical Medicine London UK.

出版信息

Campbell Syst Rev. 2019 Jul 26;15(1-2):e1027. doi: 10.1002/cl2.1027. eCollection 2019 Jun.


DOI:10.1002/cl2.1027
PMID:37131472
原文链接:https://pmc.ncbi.nlm.nih.gov/articles/PMC8356524/
Abstract

BACKGROUND: Many systematic reviews incorporate nonrandomised studies of effects, sometimes called quasi-experiments or natural experiments. However, the extent to which nonrandomised studies produce unbiased effect estimates is unclear in expectation or in practice. The usual way that systematic reviews quantify bias is through "risk of bias assessment" and indirect comparison of findings across studies using meta-analysis. A more direct, practical way to quantify the bias in nonrandomised studies is through "internal replication research", which compares the findings from nonrandomised studies with estimates from a benchmark randomised controlled trial conducted in the same population. Despite the existence of many risks of bias tools, none are conceptualised to assess comprehensively nonrandomised approaches with selection on unobservables, such as regression discontinuity designs (RDDs). The few that are conceptualised with these studies in mind do not draw on the extensive literature on internal replications (within-study comparisons) of randomised trials. OBJECTIVES: Our research objectives were as follows:Objective 1: to undertake a systematic review of nonrandomised internal study replications of international development interventions.Objective 2: to develop a risk of bias tool for RDDs, an increasingly common method used in social and economic programme evaluation. METHODS: We used the following methods to achieve our objectives.Objective 1: we searched systematically for nonrandomised internal study replications of benchmark randomised experiments of social and economic interventions in low- and middle-income countries (L&MICs). We assessed the risk of bias in benchmark randomised experiments and synthesised evidence on the relative bias effect sizes produced by benchmark and nonrandomised comparison arms.Objective 2: We used document review and expert consultation to develop further a risk of bias tool for quasi-experimental studies of interventions (ROBINS-I) for RDDs. RESULTS: Objective 1: we located 10 nonrandomised internal study replications of randomised trials in L&MICs, six of which are of RDDs and the remaining use a combination of statistical matching and regression techniques. We found that benchmark experiments used in internal replications in international development are in the main well-conducted but have "some concerns" about threats to validity, usually arising due to the methods of outcomes data collection. Most internal replication studies report on a range of different specifications for both the benchmark estimate and the nonrandomised replication estimate. We extracted and standardised 604 bias coefficient effect sizes from these studies, and present average results narratively.Objective 2: RDDs are characterised by prospective assignment of participants based on a threshold variable. Our review of the literature indicated there are two main types of RDD. The most common type of RDD is designed retrospectively in which the researcher identifies post-hoc the relationship between outcomes and a threshold variable which determines assignment to intervention at pretest. These designs usually draw on routine data collection such as administrative records or household surveys. The other, less common, type is a prospective design where the researcher is also involved in allocating participants to treatment groups from the outset. We developed a risk of bias tool for RDDs. CONCLUSIONS: Internal study replications provide the grounds on which bias assessment tools can be evidenced. We conclude that existing risk of bias tools needs to be further developed for use by Campbell collaboration authors, and there is a wide range of risk of bias tools and internal study replications to draw on in better designing these tools. We have suggested the development of a promising approach for RDD. Further work is needed on common methodologies in programme evaluation, for example on statistical matching approaches. We also highlight that broader efforts to identify all existing internal replication studies should consider more specialised systematic search strategies within particular literatures; so as to overcome a lack of systematic indexing of this evidence.

摘要

背景:许多系统评价纳入了效应的非随机研究,有时也称为准实验或自然实验。然而,非随机研究在预期或实际中产生无偏效应估计的程度尚不清楚。系统评价量化偏倚的常用方法是通过“偏倚风险评估”以及使用荟萃分析对各研究结果进行间接比较。一种更直接、实用的量化非随机研究中偏倚的方法是通过“内部重复研究”,即将非随机研究的结果与在同一人群中进行的基准随机对照试验的估计值进行比较。尽管存在许多偏倚风险工具,但没有一种被概念化为能全面评估基于不可观测因素进行选择的非随机方法,如回归断点设计(RDD)。少数考虑到这些研究的工具并未借鉴关于随机试验内部重复(研究内比较)的大量文献。 目的:我们的研究目标如下: 目标1:对国际发展干预措施的非随机内部研究重复进行系统评价。 目标2:为回归断点设计开发一种偏倚风险工具,回归断点设计是社会和经济项目评价中越来越常用的方法。 方法:我们采用以下方法来实现目标。 目标1:我们系统搜索了低收入和中等收入国家(L&MICs)社会和经济干预基准随机试验的非随机内部研究重复。我们评估了基准随机试验中的偏倚风险,并综合了关于基准组和非随机比较组产生的相对偏倚效应大小的证据。 目标2:我们通过文献回顾和专家咨询,进一步开发了一种针对回归断点设计的干预措施准实验研究的偏倚风险工具(ROBINS-I)。 结果:目标1:我们找到了低收入和中等收入国家10项随机试验中的非随机内部研究重复,其中6项是回归断点设计,其余使用统计匹配和回归技术的组合。我们发现,国际发展内部重复中使用的基准实验总体开展良好,但对有效性的威胁存在“一些担忧”*,这通常是由于结果数据收集方法导致的。大多数内部重复研究报告了基准估计值和非随机重复估计值的一系列不同规格。我们从这些研究中提取并标准化了604个偏倚系数效应大小,并以叙述方式呈现平均结果。 目标2:回归断点设计的特点是基于一个阈值变量对参与者进行前瞻性分配。我们对文献的回顾表明,回归断点设计主要有两种类型。最常见的回归断点设计类型是回顾性设计,即研究者事后确定结果与一个阈值变量之间的关系,该阈值变量在预测试时决定干预分配。这些设计通常利用常规数据收集,如行政记录或家庭调查。另一种不太常见的类型是前瞻性设计,即研究者从一开始就参与将参与者分配到治疗组。我们开发了一种回归断点设计的偏倚风险工具。 结论:内部研究重复为偏倚评估工具提供证据基础。我们得出结论,现有偏倚风险工具需要由坎贝尔协作组织的作者进一步开发,并且在更好地设计这些工具时,有广泛的偏倚风险工具和内部研究重复可供借鉴。我们提出了一种有前景的回归断点设计方法。需要对项目评价中的常用方法,例如统计匹配方法,开展进一步研究。我们还强调,为识别所有现有内部重复研究而进行的更广泛努力应考虑在特定文献中采用更专门的系统搜索策略;以克服这一证据缺乏系统索引状态的问题。

*注:原文“some concerns”直译为“一些担忧”,结合语境这里可理解为对有效性威胁方面存在一些需要关注的地方,但未明确说明担忧的具体内容,故保留原文表述。

https://cdn.ncbi.nlm.nih.gov/pmc/blobs/5bfd/8356524/10b44fd3628c/CL2-15-e1027-g001.jpg
https://cdn.ncbi.nlm.nih.gov/pmc/blobs/5bfd/8356524/042605ff4b49/CL2-15-e1027-g002.jpg
https://cdn.ncbi.nlm.nih.gov/pmc/blobs/5bfd/8356524/eb2a8ffff964/CL2-15-e1027-g003.jpg
https://cdn.ncbi.nlm.nih.gov/pmc/blobs/5bfd/8356524/10b44fd3628c/CL2-15-e1027-g001.jpg
https://cdn.ncbi.nlm.nih.gov/pmc/blobs/5bfd/8356524/042605ff4b49/CL2-15-e1027-g002.jpg
https://cdn.ncbi.nlm.nih.gov/pmc/blobs/5bfd/8356524/eb2a8ffff964/CL2-15-e1027-g003.jpg
https://cdn.ncbi.nlm.nih.gov/pmc/blobs/5bfd/8356524/10b44fd3628c/CL2-15-e1027-g001.jpg

相似文献

[1]
Within study comparisons and risk of bias in international development: Systematic review and critical appraisal.

Campbell Syst Rev. 2019-7-26

[2]
Folic acid supplementation and malaria susceptibility and severity among people taking antifolate antimalarial drugs in endemic areas.

Cochrane Database Syst Rev. 2022-2-1

[3]
School-based interventions for reducing disciplinary school exclusion: a systematic review.

Campbell Syst Rev. 2018-1-9

[4]
Recovery schools for improving behavioral and academic outcomes among students in recovery from substance use disorders: a systematic review.

Campbell Syst Rev. 2018-10-4

[5]
Deployment of personnel to military operations: impact on mental health and social functioning.

Campbell Syst Rev. 2018-6-1

[6]
The future of Cochrane Neonatal.

Early Hum Dev. 2020-11

[7]
Public sector reforms and their impact on the level of corruption: A systematic review.

Campbell Syst Rev. 2021-5-24

[8]
Impact of summer programmes on the outcomes of disadvantaged or 'at risk' young people: A systematic review.

Campbell Syst Rev. 2024-6-13

[9]
Can Non-Randomised Studies of Interventions Provide Unbiased Effect Estimates? A Systematic Review of Internal Replication Studies.

Eval Rev. 2023-6

[10]
Police-initiated diversion for youth to prevent future delinquent behavior: a systematic review.

Campbell Syst Rev. 2018-6-1

引用本文的文献

[1]
The impact of the U.S federal minimum legal drinking age law on alcohol involved firearm suicide: a regression discontinuity approach.

EClinicalMedicine. 2025-1-10

[2]
PROTOCOL: The effects of agricultural output market access interventions on agricultural, socio-economic and food and nutrition security outcomes in low- and middle-income countries: A systematic review.

Campbell Syst Rev. 2023-8-21

[3]
Editorial: Types of methods research papers in the journal .

Campbell Syst Rev. 2021-6-23

[4]
PROTOCOL: Water, sanitation and hygiene for reducing childhood mortality in low- and middle-income countries.

Campbell Syst Rev. 2021-1-19

[5]
External control arm analysis: an evaluation of propensity score approaches, G-computation, and doubly debiased machine learning.

BMC Med Res Methodol. 2022-12-28

[6]
Can Non-Randomised Studies of Interventions Provide Unbiased Effect Estimates? A Systematic Review of Internal Replication Studies.

Eval Rev. 2023-6

[7]
The Role of Stomach Infrastructures on Children's Work and Child Labour in Africa: Systematic Review.

Int J Environ Res Public Health. 2021-8-13

[8]
Study Designs to Assess Real-World Interventions to Prevent COVID-19.

Front Public Health. 2021

本文引用的文献

[1]
What are the effects of Teach For America on Math, English Language Arts, and Science outcomes of K-12 students in the USA?

Campbell Syst Rev. 2018-6-25

[2]
Reducing unemployment benefit duration to increase job finding rates: a systematic review.

Campbell Syst Rev. 2018-2-28

[3]
Do evidence summaries increase health policy-makers' use of evidence from systematic reviews? A systematic review.

Campbell Syst Rev. 2018-9-10

[4]
Regression Discontinuity Designs in Health: A Systematic Review.

Epidemiology. 2021-1

[5]
Designs of Empirical Evaluations of Nonexperimental Methods in Field Settings.

Eval Rev. 2018-4

[6]
Assessing Correspondence Between Experimental and Nonexperimental Estimates in Within-Study Comparisons.

Eval Rev. 2018-4

[7]
Assessing the Impact of a School Subsidy Program in Mexico: Using a Social Experiment to Validate a Dynamic Behavioral Model of Child Schooling and Fertility.

Am Econ Rev. 2006-12

[8]
Education as Liberation?

Economica. 2016-1

[9]
Quasi-experimental study designs series-paper 1: introduction: two historical lineages.

J Clin Epidemiol. 2017-9

[10]
Quasi-experimental study designs series-paper 7: assessing the assumptions.

J Clin Epidemiol. 2017-9

文献AI研究员

20分钟写一篇综述,助力文献阅读效率提升50倍

立即体验

用中文搜PubMed

大模型驱动的PubMed中文搜索引擎

马上搜索

推荐工具

医学文档翻译智能文献检索