Emergency Medicine, Stony Brook University Medical Center, Stony Brook, NY, USA.
Acad Emerg Med. 2011 Oct;18 Suppl 2:S65-70. doi: 10.1111/j.1553-2712.2011.01162.x.
Most multisource feedback (MSF) evaluations are performed asynchronously, with raters reflecting on the subject's behavior. Numerous studies have demonstrated poor inter-rater reliability of MSF. This may be due to cognitive biases that are inherent in such a process. We sought to determine if within- and between-rater group reliability is increased when evaluations are gathered synchronously and relate to a specific patient interaction.
This was a survey at a university emergency department (ED) of 30 emergency medicine (EM) residents. ED nurses and faculty anonymously participated in asynchronous MSF assessment of resident performance from February to April 2010 using a Web-based survey, the Emergency Medicine Humanism Scale (EM-HS). In May 2010, a second round of MSF collection was conducted in the ED. At the conclusion of patient encounters, the EM-HS was synchronously obtained from ED nurses and faculty. Evaluators were instructed to assess the resident based on the patient encounter, placing aside any preconceptions of resident performance, attitude, or behavior. Evaluators rated resident performance using a 1-9 scale ("needs improvement" to "outstanding"). The mean rating for each of the questions and the total score provided by each evaluator class was calculated for each EM resident. Differences between the asynchronous and synchronous ratings were compared with t-tests. Pearson correlations were used to measure agreement in scores within and between nurse and faculty rater groups. Correlations > 0.70 were deemed acceptable and are reported with 95% confidence intervals (CIs).
Twenty-one of 30 residents had assessments collected by both asynchronous and synchronous methods. A total of 699 Web-based (asynchronous) assessments were completed by nurses and 149 by faculty. Synchronous nurse and faculty assessments were obtained in 105 resident-patient encounters. There was no difference in faculty ratings between the MSF collection methods. Nurses assigned slightly (but significantly) higher ratings during synchronous collection. Correlation of the total MSF score between asynchronous and synchronous feedback collection methods within the faculty rater group was poor (0.18, 95% CI = -0.22 to 0.60). Correlation of the total MSF score between asynchronous and synchronous feedback collection methods within nurse rater groups was moderate (0.63, 95% CI = 0.27 to 0.83). Correlations between faculty-nurse rater groups for the total MSF collected asynchronously and synchronously were moderate (0.39, 95% CI = -0.05 to 0.7; and 0.44, 95% CI 0.01 to 0.73, respectively).
Synchronous collection of MSF did not provide clinically different EM-HS scores within rater groups and did not result in improved correlations. Our small, single-center study supports asynchronous collection of MSF.
大多数多源反馈（MSF）评估都是异步进行的，评分者会对被评估者的行为进行反思。许多研究表明 MSF 的评分者间可靠性较差。这可能是由于这种评估过程中固有的认知偏差造成的。我们试图确定当评估是同步进行并与特定患者交互相关联时，评分者内和评分者间的可靠性是否会提高。
这是一项在大学急诊科（ED）进行的调查，涉及 30 名急诊医学（EM）住院医师。ED 护士和教师匿名参与了 2010 年 2 月至 4 月期间使用基于网络的调查，即急诊医学人文量表（EM-HS）对住院医师表现进行的异步 MSF 评估。2010 年 5 月，在 ED 中进行了第二轮 MSF 收集。在患者就诊结束时，ED 护士和教师同步获得 EM-HS。评估者被指示根据患者的就诊情况对住院医师进行评估，将对住院医师表现、态度或行为的任何先入为主的观念放在一边。评估者使用 1-9 分制（“需要改进”到“出色”）对住院医师的表现进行评分。为每位 EM 住院医师计算了每个问题的平均评分和每个评估者类别提供的总分。使用 t 检验比较异步和同步评分之间的差异。Pearson 相关系数用于测量护士和教师评分者组内和组间评分的一致性。> 0.70 的相关性被认为是可接受的，并报告了 95%置信区间（CI）。
30 名住院医师中有 21 名接受了异步和同步方法的评估。共有 699 名护士和 149 名教师完成了基于网络的（异步）评估。在 105 次住院医师-患者就诊中获得了同步护士和教师的评估。教师评分在两种 MSF 收集方法之间没有差异。护士在同步收集时的评分略高（但差异有统计学意义）。教师评分者组中异步和同步反馈收集方法之间的总 MSF 评分之间的相关性较差（0.18，95%CI=-0.22 至 0.60）。护士评分者组中异步和同步反馈收集方法之间的总 MSF 评分之间的相关性为中度（0.63，95%CI=0.27 至 0.83）。异步和同步收集的总 MSF 评分在教师-护士评分者组之间的相关性为中度（分别为 0.39，95%CI=-0.05 至 0.7 和 0.44，95%CI 0.01 至 0.73）。
在评分者内部，同步收集 MSF 并没有提供临床差异的 EM-HS 评分，也没有提高相关性。我们的小型单中心研究支持异步收集 MSF。
