Makkar Steve R, Williamson Anna, Turner Tari, Redman Sally, Louviere Jordan
The Sax Institute, Level 13, Building 10, 235 Jones Street, Ultimo, New South Wales, 2007, Australia.
World Vision Australia, 1 Vision Drive, Burwood East, Victoria, 3151, Australia.
Health Res Policy Syst. 2015 Apr 26;13:22. doi: 10.1186/s12961-015-0013-z.
Effective use of research to inform policymaking can be strengthened by policymakers undertaking various research engagement actions (e.g., accessing, appraising, and applying research). Consequently, we developed a thorough measurement and scoring tool to assess whether and how policymakers undertook research engagement actions in the development of a policy document. This scoring tool breaks down each research engagement action into its key 'subactions' like a checklist. The primary aim was to develop the scoring tool further so that it assigned appropriate scores to each subaction based on its effectiveness for achieving evidence-informed policymaking. To establish the relative effectiveness of these subactions, we conducted a conjoint analysis, which was used to elicit the opinions and preferences of knowledge translation experts.
Fifty-four knowledge translation experts were recruited to undertake six choice surveys. Respondents were exposed to combinations of research engagement subactions called 'profiles', and rated on a 1-9 scale whether each profile represented a limited (1-3), moderate (4-6), or extensive (7-9) example of each research engagement action. Generalised estimating equations were used to analyse respondents' choice data, where a utility coefficient was calculated for each subaction. A large utility coefficient indicates that a subaction was influential in guiding experts' ratings of extensive engagement with research.
The calculated utilities were used as the points assigned to the subactions in the scoring system. The following subactions yielded the largest utilities and were regarded as the most important components of engaging with research: searching academic literature databases, obtaining systematic reviews and peer-reviewed research, appraising relevance by verifying its applicability to the policy context, appraising quality by evaluating the validity of the method and conclusions, engaging in thorough collaborations with researchers, and undertaking formal research projects to inform the policy in question.
We have generated an empirically-derived and context-sensitive method of measuring and scoring the extent to which policymakers engaged with research to inform policy development. The scoring system can be used by organisations to quantify staff research engagement actions and thus provide them with insights into what types of training, systems, and tools might improve their staff's research use capacity.
政策制定者采取各种研究参与行动（如获取、评估和应用研究成果），有助于加强研究成果在政策制定中的有效应用。因此，我们开发了一种全面的测量和评分工具，以评估政策制定者在制定政策文件时是否以及如何采取研究参与行动。该评分工具将每项研究参与行动细分为关键的“子行动”，类似于一份清单。主要目的是进一步完善该评分工具，以便根据其对实现循证决策的有效性为每个子行动分配适当的分数。为了确定这些子行动的相对有效性，我们进行了一项联合分析，用于征求知识转化专家的意见和偏好。
招募了54名知识转化专家进行六项选择调查。受访者接触到称为“概况”的研究参与子行动组合，并以1-9分的尺度对每个概况代表的每项研究参与行动是有限（1-3）、中等（4-6）还是广泛（7-9）的示例进行评分。使用广义估计方程分析受访者的选择数据，为每个子行动计算一个效用系数。效用系数越大，表明该子行动在指导专家对广泛参与研究的评分方面越有影响力。
计算出的效用被用作评分系统中分配给子行动的分数。以下子行动产生了最大的效用，被视为参与研究的最重要组成部分：搜索学术文献数据库、获取系统评价和同行评审研究、通过验证其对政策背景的适用性来评估相关性、通过评估方法和结论的有效性来评估质量、与研究人员进行深入合作以及开展正式研究项目以为相关政策提供信息。
我们生成了一种基于实证且因地制宜的方法，用于衡量和评分政策制定者在为政策制定提供信息时参与研究的程度。各组织可以使用该评分系统来量化员工的研究参与行动，从而使他们了解哪些类型的培训、系统和工具可能提高员工的研究应用能力。
