SciPinion LLC., Bozeman, MT 59715, USA.
Ted Simon LLC, Winston, GA 30187, USA.
Regul Toxicol Pharmacol. 2019 Apr;103:73-85. doi: 10.1016/j.yrtph.2019.01.003. Epub 2019 Jan 8.
Science peer review plays an important role in the advancement and acceptance of scientific information, particularly when used to support decision-making. A model for science peer review is proposed here using a large, multi-tiered case study to engage a broader segment of the scientific community to support decision making on science matters, and to incorporate many of the design advantages of the two common forms of peer review (journal peer review, science advisory panels). This peer review consisted of a two-tiered structure consisting of seven panels (five review panels in Tier 1, two review panels in Tier 2), which focused on safety data for a modified risk tobacco product (MRTP). Experts from all over the world were invited to apply to one or more positions on seven peer review panels. 66 peer reviewers were selected from available applicants using objective metrics of their expertise, and for some panels based upon a consideration of panel diversity with respect to demographic parameters (e.g., geographic region, sector of employment, years of experience). All peer reviewers participated anonymously in which a third-party auditor was used to provide independent verification of their expertise. Peer reviewers were provided electronic links to all review material which included access to publications, reports, omics data, and histopathology slides, with topic-specific panels focusing on topic-specific components of the review package. Peer reviews consisted either of single-round, or multi-round (e.g., modified Delphi) format. Peer reviewer responses to the charge questions were collected via an online survey system, and were assembled into a database. Responses in the database were subject to analyses to assess the degree of favorability (i.e., supportive of the review material), degree of consensus, reproducibility of replicate panels, hidden sources of bias, and outlier response patterns. Conclusions: By careful consideration of science peer review design elements we have shown that: 1) panel participation can be broadened to include scientists who would otherwise not participate; 2) panel diversity can be managed in an unbiased manner without adverse impacts to panel expertise; 3) results obtained from independent concurrent panels are shown to be reproducible; and 4) there are benefits of collecting input from expert panels via a structured format (i.e., survey) to support characterization of consensus, identification of hidden sources of bias, and identification of potential outlier participants.
科学同行评议在推动和接受科学信息方面发挥着重要作用，尤其是在用于支持决策时。这里提出了一种科学同行评议模式，该模式使用大型多层次案例研究来吸引更广泛的科学界参与，以支持科学事务的决策，并结合同行评议的两种常见形式（期刊同行评议、科学顾问小组）的许多设计优势。这种同行评议由两层结构组成，共七个小组（第一层五个小组，第二层两个小组），重点关注改良风险烟草制品的安全数据。邀请来自世界各地的专家申请七个同行评议小组的一个或多个职位。根据其专业知识的客观指标，从可用申请人中选择了 66 名同行评审员，对于某些小组，还根据人口参数（例如地理区域、就业部门、工作经验年限）考虑小组多样性来选择。所有同行评审员都匿名参与，第三方审计员被用来独立验证他们的专业知识。同行评审员可以通过电子链接访问所有评审材料，包括访问出版物、报告、组学数据和组织病理学幻灯片，主题特定小组专注于评审包的特定主题组件。同行评审由单轮或多轮（例如，改良 Delphi）格式组成。同行评审员对任务问题的回复通过在线调查系统收集，并汇编成数据库。数据库中的回复会进行分析，以评估赞成度（即，对评审材料的支持程度）、一致性程度、重复小组的可重复性、隐藏的偏见来源以及异常反应模式。结论：通过仔细考虑科学同行评议设计要素，我们表明：1）可以扩大小组参与范围，包括否则不会参与的科学家；2）可以以公正的方式管理小组多样性，而不会对小组专业知识产生不利影响；3）独立并行小组获得的结果是可重复的；4）通过结构化格式（即调查）从专家小组收集输入可以更好地支持共识的描述、隐藏偏见来源的识别以及潜在异常参与者的识别。
