Shershneva Marianna B, Larrison Christopher, Robertson Sheila, Speight Mike
Office of Continuing Professional Development in Medicine and Public Health, University of Wisconsin-Madison, USA.
J Contin Educ Health Prof. 2011 Fall;31 Suppl 1:S28-36. doi: 10.1002/chp.20146.
Although evaluating at multiple outcome levels has been proposed for continuing education activities and programs, it is a complex undertaking and is not done routinely, especially in collaborative, multicomponent programs. This article reports on strategies used and results obtained in an evaluation project that examined multiple outcomes of a US-based collaborative, multicomponent smoking cessation educational program for clinicians.
Evaluation was organized conceptually around the 6 levels of an outcomes-based evaluation model and was conducted using registration data, postactivity evaluations, clinical vignettes with questions assessing knowledge and competence in participants and in a comparison group, a commitment to change approach, data from patient charts to assess clinician compliance on 8 performance measures, and tobacco cessation rates. Additional methods included a success case method study of 9 practices participating in performance improvement (PI) activities and assessment of partner collaboration using a written survey and interviews.
The program reached more than 43,000 clinicians who participated in a variety of activities. Participants indicated a high level of satisfaction with the program's educational activities and demonstrated higher scores than a comparison group on 6 of 7 competencies. The majority of participants who responded to commitment to change questions reported intended and implemented practice changes consistent with desired outcomes. Performance outcomes of 3 PI activities varied, with greater improvements observed in 1 activity (9.0% to 36.2% improvement across 8 measures). Lower performance outcomes, but a smoking quit rate of 46.8%, was observed in 2 other PI activities.
The program had an overall positive impact on the measured variables for clinicians and their patients. Use of 1 outcomes assessment framework acceptable to all members of the collaborative, common measures and evaluation techniques, and centralized data repositories contributed to the success of the program evaluation reported here and is recommended to others who are considering a collaborative program evaluation.
尽管有人提议在继续教育活动和项目中从多个结果层面进行评估，但这是一项复杂的工作，并未常规开展，尤其是在协作性、多组成部分的项目中。本文报告了一个评估项目所采用的策略及取得的成果，该项目考察了美国一个针对临床医生的协作性、多组成部分戒烟教育项目的多个结果。
评估在概念上围绕基于结果的评估模型的6个层面进行组织，并利用注册数据、活动后评估、带有评估参与者及对照组知识和能力问题的临床案例、变革承诺方法、来自患者病历的数据以评估临床医生在8项绩效指标上的依从性以及戒烟率。其他方法包括对参与绩效改进（PI）活动的9个医疗机构进行成功案例方法研究，以及使用书面调查和访谈评估合作伙伴协作情况。
该项目覆盖了超过43000名参与各种活动的临床医生。参与者对项目的教育活动满意度很高，并且在7项能力中的6项上的得分高于对照组。对变革承诺问题做出回应的大多数参与者报告称，他们打算并实施了与预期结果相符的实践变革。3项PI活动的绩效结果各不相同，在1项活动中观察到了更大的改进（8项指标的改进幅度为9.0%至36.2%）。在另外2项PI活动中，绩效结果较低，但戒烟率为46.8%。
该项目对临床医生及其患者的测量变量产生了总体积极影响。使用协作各方所有成员都认可的一个结果评估框架、通用的测量方法和评估技术以及集中的数据存储库，促成了本文所报告的项目评估的成功，并推荐给其他正在考虑进行协作项目评估的人。
