Guldbrand Nielsen Dorte, Jensen Signe Lichtenstein, O'Neill Lotte
Department of Cardiology, Aarhus University Hospital, Aarhus, Denmark.
Center for Medical Education, Aarhus University, Aarhus, Denmark.
BMC Med Educ. 2015 Feb 1;15:9. doi: 10.1186/s12909-015-0294-5.
Transthoracic echocardiography (TTE) is a widely used cardiac imaging technique that all cardiologists should be able to perform competently. Traditionally, TTE competence has been assessed by unstructured observation or in test situations separated from daily clinical practice. An instrument for assessment of clinical TTE technical proficiency including a global rating score and a checklist score has previously shown reliability and validity in a standardised setting. As clinical test situations typically have several sources of error giving rise to variance in scores, a more thorough examination of the generalizability of the assessment instrument is needed.
Nine physicians performed a TTE scan on the same three patients. Then, two raters rated all 27 TTE scans using the TTE technical assessment instrument in a fully crossed, all random generalizability study. Estimated variance components were calculated for both the global rating and checklist scores. Finally, dependability (phi) coefficients were also calculated for both outcomes in a decision study.
For global rating scores, 66.6% of score variance can be ascribed to true differences in performance. For checklist scores this was 88.8%. The difference was primarily due to physician-rater interaction. Four random cases rated by one random rater resulted in a phi value of 0.81 for global ratings and two random cases rated by one random rater showed a phi value of 0.92 for checklist scores.
Using the TTE checklist as opposed to the TTE global rating score had the effect of minimising the largest source of error variance in test scores. Two cases rated by one rater using the TTE checklist are sufficiently reliable for high stakes examinations. As global rating is less time consuming it could be considered performing four global rating assessments in addition to the checklist assessments to account for both reliability and content validity of the assessment.
经胸超声心动图（TTE）是一种广泛应用的心脏成像技术，所有心脏病专家都应能够熟练操作。传统上，TTE操作能力是通过非结构化观察或在与日常临床实践分离的测试环境中进行评估的。一种用于评估临床TTE技术熟练程度的工具，包括整体评分和清单评分，此前已在标准化环境中显示出可靠性和有效性。由于临床测试环境通常有多个误差来源导致分数差异，因此需要对评估工具的可推广性进行更全面的检查。
九名医生对相同的三名患者进行TTE扫描。然后，两名评估者在一项完全交叉、全随机的可推广性研究中使用TTE技术评估工具对所有27次TTE扫描进行评分。计算了整体评分和清单评分的估计方差分量。最后，在一项决策研究中还计算了两种结果的可靠性（phi）系数。
对于整体评分，66.6%的分数差异可归因于表现上的真正差异。对于清单评分，这一比例为88.8%。差异主要是由于医生与评估者之间的相互作用。一名随机评估者对四个随机病例进行评分，整体评分的phi值为0.81，一名随机评估者对两个随机病例进行评分，清单评分的phi值为0.92。
使用TTE清单而非TTE整体评分有减少测试分数中最大误差方差来源的效果。一名评估者使用TTE清单对两个病例进行评分对于高风险考试来说足够可靠。由于整体评分耗时较少，可以考虑在清单评估之外再进行四次整体评分评估，以兼顾评估的可靠性和内容效度。
