Mckay Virginia, Zamantakis Alithia, Pachicano Ana Michaela, Merle James, Purrier Morgan, Swan McKenzie, Li Dennis, Mustanski Brian, Smith Justin D, Hirschhorn Lisa, Benbow Nanette
Washington University in Saint Louis.
Northwestern University.
Res Sq. 2024 Feb 29:rs.3.rs-3979631. doi: 10.21203/rs.3.rs-3979631/v1.
There are no criteria specifically for evaluating the quality of implementation research and recommend implementation strategies likely to have impact to practitioners. We describe the development and application of the Best Practices Rubric, a set of criteria to evaluate the evidence supporting implementation strategies, in the context of HIV.
We developed the Best Practices Rubric from 2022-2023 in three phases. (1) We purposively selected and recruited by email participants representing a mix of expertise in HIV service delivery, quality improvement, and implementation science. We developed a draft rubric and criteria based on a literature review and key informant interviews. (2) The rubric was then informed and revised through two e-Delphi rounds using a survey delivered online through Qualtrics. The first and second round Delphi surveys consisted of 71 and 52 open and close-ended questions, respectively, asking participants to evaluate, confirm, and make suggestions on different aspects of the rubric. After each survey round, data were analyzed and synthesized as appropriate, and the rubric and criteria were revised. (3) We then applied the rubric to a set of research studies assessing 18 implementation strategies designed to promote the adoption and uptake of pre-exposure prophylaxis, an HIV prevention medication, to assess reliable application of the rubric and criteria.
Our initial literature review yielded existing rubrics and criteria for evaluating intervention-level evidence. For a strategy-level rubric, additions emerged from interviews, for example, a need to consider the context and specification of strategies. Revisions were made after both Delphi rounds resulting in the confirmation of five evaluation domains - research design, implementation outcomes, limitations and rigor, strategy specification, and equity - and four evidence levels - best practice, promising practice, more evidence needed, and harmful practices. For most domains, criteria were specified at each evidence level. After an initial pilot round to develop an application process and provide training, we achieved 98% reliability when applying the criteria to 18 implementation strategies.
We developed a rubric to evaluate the evidence supporting implementation strategies for HIV services. Although the rubric is specific to HIV, this tool is adaptable for evaluating strategies in other health areas.
目前尚无专门用于评估实施研究质量并向从业者推荐可能产生影响的实施策略的标准。我们描述了最佳实践评分标准的制定和应用，这是一套在艾滋病病毒（HIV）背景下评估支持实施策略证据的标准。
我们在2022年至2023年分三个阶段制定了最佳实践评分标准。（1）我们通过电子邮件有目的地挑选并招募了代表HIV服务提供、质量改进和实施科学等多方面专业知识的参与者。我们基于文献综述和关键信息访谈制定了评分标准草案和准则。（2）然后通过两轮电子德尔菲法对评分标准进行完善和修订，使用通过Qualtrics在线发放的调查问卷。第一轮和第二轮德尔菲调查分别包含71个和52个开放式和封闭式问题，要求参与者对评分标准的不同方面进行评估、确认并提出建议。每轮调查后，对数据进行适当分析和综合，并对评分标准和准则进行修订。（3）然后我们将该评分标准应用于一组评估18种实施策略的研究，这些策略旨在促进暴露前预防（一种HIV预防药物）的采用和使用，以评估评分标准和准则的可靠应用。
我们最初的文献综述得出了用于评估干预层面证据的现有评分标准和准则。对于策略层面的评分标准，通过访谈增加了一些内容，例如需要考虑策略的背景和具体说明。两轮德尔菲法之后都进行了修订，最终确定了五个评估领域——研究设计、实施结果、局限性和严谨性、策略说明以及公平性——以及四个证据级别——最佳实践、有前景的实践、需要更多证据以及有害实践。对于大多数领域，在每个证据级别都规定了相应的标准。在进行了一轮初步试点以制定应用流程并提供培训后，我们将该标准应用于18种实施策略时，可靠性达到了98%。
我们制定了一个评分标准来评估支持HIV服务实施策略的证据。尽管该评分标准特定于HIV，但此工具适用于评估其他健康领域的策略。
