Sampaio Carvalho Hervaldo, Barros Melo Maria do Carmo, Kakehasi Fabiana Maria, Menezes Ferri Liu Priscila, Gontijo Minafra Silveira Santos Fernanda, de Melo Martins Ricardo Luiz, Vale de Castro Monteiro Marilene, Chaves Campos Marcelo Esteves, Esteves Roberto Zonato, Belloni Torsani Matheus, Lopes Calvo Tiberio Iolanda de Fátima, Inca Atahualpa Urdiales Akihito, De Marco Novellino Andrea Mora, Viana Zuza Diniz Rosiane, Ferreira Moura Helena, de Vilhena Toledo Maria Alice, Machado Saldanha Célia Regina, Pedrosa Ribeiro Alves Oliveira Andrea, Zen Tempski Patricia, de Souza Santos Itamar, de Arruda Martins Milton
Internal Medicine/Cardiology, Medical School, Brasília University, Brasília, Brazil.
Department of Pediatrics, Federal University of Minas Gerais, Belo Horizonte, Brazil.
BMC Med Educ. 2025 Jan 13;25(1):54. doi: 10.1186/s12909-024-06631-4.
The COVID-19 pandemic has led to the popularization of information and communication technology in medical education. This study aimed to compare the objective structured clinical examination (OSCE) and the virtual objective structured clinical examination (vOSCE), based on expert opinion, as tools for assessing the competencies needed under medical curriculum guidelines in Brazil.
In this multicenter study, the suitability levels of the OSCE and vOSCE for assessing the competencies needed under the Brazilian National Curriculum Guidelines (DCNs) were compared. The DCNs encompass five groups of domains, namely, health education; general; health management; public health needs; and individual health needs. Sixteen teaching experts served as judges. They responded to a questionnaire with the provided Likert scores indicating the ability of the OSCE and vOSCE to assess each type of competency. Virtual meetings were held via the Delphi method and a focus group-based approach to enable the experts to reach a consensus.
A total of 200 items included in the guidelines (100 for the OSCE and 100 for the vOSCE) were evaluated across a total of 3,200 responses. Analysis via the Likert scale revealed a high proportion of positive evaluations for the use of the vOSCE, with kappa values ranging from 0.4 to 0.72. The values of the absolute agreement (positive view) between examiners and equivalence with the kappa statistic for the adoption of the vOSCE ranged from 0.38 to 0.72. Via the use of virtual meetings (Delphi and focus groups), consensus was reached regarding the capacity of the OSCE and vOSCE to evaluate 97% and 92% of these items, respectively.
The vOSCE can be employed to assess relevant competencies. However, it faces limitations regarding 8% (8/100) of the items, and some items (3%) cannot be assessed via either method. The difference in results between the two assessment methodologies (OSCE and vOSCE) is 5%. It is necessary to invest in the creation of instruments that can be used to apply the vOSCE and to conduct cost‒benefit analysis of its broader application in health education institutions. We conclude that the vOSCE is an effective tool for assessing most competencies of medical professionals and students required under Brazilian guidelines.
新型冠状病毒肺炎疫情推动了信息通信技术在医学教育中的普及。本研究旨在根据专家意见，比较客观结构化临床考试（OSCE）和虚拟客观结构化临床考试（vOSCE），作为评估巴西医学课程指南所要求能力的工具。
在这项多中心研究中，比较了OSCE和vOSCE对评估巴西国家课程指南（DCNs）所要求能力的适用性水平。DCNs涵盖五组领域，即健康教育；一般领域；健康管理；公共卫生需求；以及个人健康需求。16名教学专家担任评委。他们对一份问卷做出回应，问卷提供了李克特评分，表明OSCE和vOSCE评估每种能力类型的能力。通过德尔菲法和基于焦点小组的方法举行虚拟会议，以使专家们达成共识。
指南中总共200项内容（OSCE和vOSCE各100项）在总共3200份回复中得到评估。通过李克特量表分析发现，对vOSCE使用的积极评价比例很高，kappa值范围为0.4至0.72。考官之间的绝对一致性（积极观点）值以及采用vOSCE的kappa统计量的等效性值范围为0.38至0.72。通过使用虚拟会议（德尔菲法和焦点小组），就OSCE和vOSCE分别评估这些项目的97%和92%的能力达成了共识。
vOSCE可用于评估相关能力。然而，它在8%（8/100）的项目上存在局限性，并且一些项目（3%）无法通过任何一种方法进行评估。两种评估方法（OSCE和vOSCE）的结果差异为5%。有必要投资创建可用于应用vOSCE的工具，并对其在健康教育机构中的更广泛应用进行成本效益分析。我们得出结论，vOSCE是评估巴西指南所要求的医学专业人员和学生的大多数能力的有效工具。
