一种用于住院医师的基于德语能力的多源反馈工具：开发与效度证据

A german-language competency-based multisource feedback instrument for residents: development and validity evidence.

作者信息

Hennel Eva K, Subotic Ulrike, Berendonk Christoph, Stricker Daniel, Harendza Sigrid, Huwendiek Sören

机构信息

Department for Assessment and Evaluation (AAE), Institute for Medical Education, University of Bern, Mittelstrasse 43, 3012, Bern, Switzerland.

University Children's Hospital Basel, Spitalstrasse 33, 4056, Basel, Switzerland.

出版信息

BMC Med Educ. 2020 Oct 12;20(1):357. doi: 10.1186/s12909-020-02259-2.

DOI:10.1186/s12909-020-02259-2

PMID:33046060

原文链接:https://pmc.ncbi.nlm.nih.gov/articles/PMC7552497/

Abstract

BACKGROUND

In medical settings, multisource feedback (MSF) is a recognised method of formative assessment. It collects feedback on a doctor's performance from several perspectives in the form of questionnaires. Yet, no validated MSF questionnaire has been publicly available in German. Thus, we aimed to develop a German MSF questionnaire based on the CanMEDS roles and to investigate the evidence of its validity.

METHODS

We developed a competency-based MSF questionnaire in German, informed by the literature and expert input. Four sources of validity evidence were investigated: (i) Content was examined based on MSF literature, blueprints of competency, and expert-team discussions. (ii) The response process was supported by analysis of a think-aloud study, narrative comments, "unable to comment" ratings and evaluation data. (iii) The internal structure was assessed by exploratory factor analysis, and inter-rater reliability by generalisability analysis. Data were collected during two runs of MSF, in which 47 residents were evaluated once (first run) or several times (second and third run) on 81 occasions of MSF. (iv) To investigate consequences, we analysed the residents' learning goals and the progress as reported via MSF.

RESULTS

Our resulting MSF questionnaire (MSF-RG) consists of 15 items and one global rating, which are each rated on a scale and accompanied by a field for narrative comments and cover a construct of a physician's competence. Additionally, there are five open questions for further suggestions. Investigation of validity evidence revealed that: (i) The expert group agreed that the content comprehensively addresses clinical competence; (ii) The response processes indicated that the questions are understood as intended and supported the acceptance and usability; (iii) For the second run, factor analysis showed a one-factor solution, a Cronbach's alpha of 0.951 and an inter-rater reliability of 0.797 with 12 raters; (iv) There are indications that residents benefitted, considering their individual learning goals and based on their ratings reported via MSF itself.

CONCLUSIONS

To support residency training with multisource feedback, we developed a German MSF questionnaire (MSF-RG), which is supported by four sources of validity evidence. This MSF questionnaire may be useful to implement MSF in residency training in German-speaking regions.

摘要

背景

在医疗环境中，多源反馈（MSF）是一种公认的形成性评估方法。它通过问卷调查的形式从多个角度收集关于医生表现的反馈。然而，目前尚无经过验证的德文版MSF问卷公开可用。因此，我们旨在基于加拿大医学教育专业人员角色（CanMEDS）开发一份德文版MSF问卷，并调查其有效性证据。

方法

我们在文献和专家意见的基础上，开发了一份基于能力的德文版MSF问卷。研究了四个有效性证据来源：（i）根据MSF文献、能力蓝图和专家团队讨论来审查内容。（ii）通过有声思维研究分析、叙述性评论、“无法评论”评分和评估数据来支持应答过程。（iii）通过探索性因素分析评估内部结构，并通过概化分析评估评分者间信度。在两次MSF实施过程中收集数据，在这两次过程中，47名住院医师在81次MSF评估中接受了一次（第一次实施）或多次（第二次和第三次实施）评估。（iv）为了调查结果，我们分析了住院医师通过MSF报告的学习目标和进展情况。

结果

我们最终得到的MSF问卷（MSF-RG）由15个项目和一个总体评分组成，每个项目都按量表进行评分，并伴有一个叙述性评论字段，涵盖了医生能力的一个结构。此外，还有五个开放性问题以获取进一步建议。有效性证据调查表明：（i）专家组一致认为内容全面涵盖了临床能力；（ii）应答过程表明问题按预期被理解，并支持问卷的可接受性和可用性；（iii）在第二次实施中，因素分析显示为单因素解决方案，Cronbach's alpha系数为0.951，12名评分者的评分者间信度为0.797；（iv）有迹象表明，考虑到住院医师的个人学习目标并基于他们通过MSF本身报告的评分，他们从中受益。