Faculty of Psychology and Educational Sciences, University of Geneva, Geneva, Switzerland.
Center for Reproducible Science (CRS), University of Zurich, Zurich, Switzerland.
F1000Res. 2024 Jun 3;12:588. doi: 10.12688/f1000research.129814.3. eCollection 2023.
The quality of COVID-19 preprints should be considered with great care, as their contents can influence public policy. Surprisingly little has been done to calibrate the public's evaluation of preprints and their contents. The PRECHECK project aimed to generate a tool to teach and guide scientifically literate non-experts to critically evaluate preprints, on COVID-19 and beyond.
To create a checklist, we applied a four-step procedure consisting of an initial internal review, an external review by a pool of experts (methodologists, meta-researchers/experts on preprints, journal editors, and science journalists), a final internal review, and a Preliminary implementation stage. For the external review step, experts rated the relevance of each element of the checklist on five-point Likert scales, and provided written feedback. After each internal review round, we applied the checklist on a small set of high-quality preprints from an online list of milestone research works on COVID-19 and low-quality preprints, which were eventually retracted, to verify whether the checklist can discriminate between the two categories.
At the external review step, 26 of the 54 contacted experts responded. The final checklist contained four elements (Research question, study type, transparency and integrity, and limitations), with 'superficial' and 'deep' evaluation levels. When using both levels, the checklist was effective at discriminating a small set of high- and low-quality preprints. Its usability for assessment and discussion of preprints was confirmed in workshops with Bachelors students in Psychology and Medicine, and science journalists.
We created a simple, easy-to-use tool for helping scientifically literate non-experts navigate preprints with a critical mind and facilitate discussions within, for example, a beginner-level lecture on research methods. We believe that our checklist has potential to help guide decisions about the quality of preprints on COVID-19 in our target audience and that this extends beyond COVID-19.
COVID-19 预印本的质量需要谨慎考虑，因为其内容可能会影响公共政策。令人惊讶的是，人们很少努力校准公众对预印本及其内容的评估。PRECHECK 项目旨在开发一种工具，以教导和指导有科学素养的非专业人士批判性地评估预印本，无论是关于 COVID-19 还是其他领域。
为了创建清单，我们采用了四步程序，包括初步内部审查、由一组专家（方法学家、预印本元研究/专家、期刊编辑和科学记者）进行的外部审查、最终内部审查和初步实施阶段。在外部审查步骤中，专家们对清单的每个要素进行了五分制李克特量表的相关性评估，并提供了书面反馈。在每次内部审查后，我们使用清单对来自 COVID-19 里程碑研究工作在线列表的一小部分高质量预印本和最终撤回的低质量预印本进行了评估，以验证清单是否可以区分这两类。
在外部审查步骤中，联系的 54 位专家中有 26 位做出了回应。最终清单包含四个要素（研究问题、研究类型、透明度和完整性以及局限性），具有“表面”和“深入”评估层次。当使用这两个层次时，清单可以有效地区分一小部分高质量和低质量的预印本。在与心理学和医学学士学生以及科学记者的研讨会上，验证了其对预印本评估和讨论的可用性。
我们创建了一个简单易用的工具，帮助有科学素养的非专业人士批判性地浏览预印本，并促进讨论，例如，在研究方法的入门级讲座中。我们相信，我们的清单有可能帮助我们的目标受众中的决策者评估 COVID-19 预印本的质量，并且这种方法可以扩展到 COVID-19 之外。
