S. Ginsburg is professor of medicine, Department of Medicine, Faculty of Medicine, University of Toronto, scientist, Wilson Centre for Research in Education, University Health Network, University of Toronto, Toronto, Ontario, Canada, and Canada Research Chair in Health Professions Education; ORCID: http://orcid.org/0000-0002-4595-6650.
A. Gingerich is assistant professor, Northern Medical Program, University of Northern British Columbia, Prince George, British Columbia, Canada; ORCID: https://orcid.org/0000-0001-5765-3975.
Acad Med. 2020 Nov;95(11S Association of American Medical Colleges Learn Serve Lead: Proceedings of the 59th Annual Research in Medical Education Presentations):S81-S88. doi: 10.1097/ACM.0000000000003643.
Written comments are gaining traction as robust sources of assessment data. Compared with the structure of numeric scales, what faculty choose to write is ad hoc, leading to idiosyncratic differences in what is recorded. This study offers exploration of what aspects of writing styles are determined by the faculty offering comment and what aspects are determined by the trainee being commented upon.
The authors compiled in-training evaluation report comment data, generated from 2012 to 2015 by 4 large North American Internal Medicine training programs. The Linguistic Index and Word Count (LIWC) was used to categorize and quantify the language contained. Generalizability theory was used to determine whether faculty could be reliably discriminated from one another based on writing style. Correlations and ANOVAs were used to determine what styles were related to faculty or trainee demographics.
Datasets contained 23-142 faculty who provided 549-2,666 assessments on 161-989 trainees. Faculty could easily be discriminated from one another using a variety of LIWC metrics including word count, words per sentence, and the use of "clout" words. These patterns appeared person specific and did not reflect demographic factors such as gender or rank. These metrics were similarly not consistently associated with trainee factors such as postgraduate year or gender.
Faculty seem to have detectable writing styles that are relatively stable across the trainees they assess, which may represent an under-recognized source of construct irrelevance. If written comments are to meaningfully contribute to decision making, we need to understand and account for idiosyncratic writing styles.
书面评论作为强有力的评估数据来源越来越受到重视。与数字量表的结构相比，教师选择写什么是特别的，这导致记录的内容存在独特的差异。本研究探讨了教师提供的评论的哪些方面以及被评论的学员的哪些方面决定了写作风格。
作者汇编了 2012 年至 2015 年 4 个大型北美内科培训项目的住院医师评估报告的评论数据。使用语言指数和单词计数（LIWC）对包含的语言进行分类和量化。使用概化理论来确定教师是否可以根据写作风格可靠地区分。使用相关性和方差分析来确定哪些风格与教师或学员的人口统计学特征相关。
数据集包含 23-142 名教师，他们对 161-989 名学员的 549-2666 次评估提供了 549-2666 次评估。教师可以使用各种 LIWC 指标（包括字数、每句话的字数和“影响力”词汇的使用）轻松区分彼此。这些模式似乎是特定于个人的，并不反映性别或职级等人口统计学因素。这些指标也与学员的因素（如研究生年级或性别）不一致。
教师似乎有可察觉的写作风格，在他们评估的学员中相对稳定，这可能代表一种未被认识到的结构不相关的来源。如果书面评论要对决策有意义地做出贡献，我们需要理解和考虑独特的写作风格。
