Department of Pediatrics, Center for Healthcare Policy and Research, University of California Davis School of Medicine, Sacramento, California 95817, USA.
Acad Med. 2013 Aug;88(8):1107-15. doi: 10.1097/ACM.0b013e31829a249a.
The annual U.S. News & World Report (USN&WR) Primary Care Medical School (PCMS) ranking attracts considerable attention, but its measurement properties have not been published. The authors examined the short-term stability of the PCMS ranking and the PCMS score from which it derives, along with the short-term spread of schools' rankings.
The authors employed published data and methods to reconstruct the 2009-2012 PCMS scores and rankings. They used mixed-effects models to assess the within-school, between-year reliability (short-term stability) of the PCMS score and ranking, yielding intraclass correlation coefficients (ICCs). They defined short-term spread as the median within-school range in ranking across the four-year study period.
Reconstructed PCMS scores correlated highly with published scores all four years (Pearson correlations≥98.9%). Most schools' mean annual PCMS scores were tightly clustered near the center of the score distribution. ICCs for the PCMS score and ranking were, respectively, 94% and 90%. The median difference between the best and worst ranking over the study period was 4 for the 18 schools with an average annual ranking of 1 to 20, and 17 for the other 89 schools (P<.001, Kruskal-Wallis test).
The short-term stability of the USN&WR PCMS score and ranking were reasonably good. However, the short-term spread in PCMS rankings was large, particularly among schools with mean annual rankings below the top 20. The variability is greater than could be plausibly attributed to actual changes in training quality. These findings raise questions regarding the ranking's validity and usefulness.
美国新闻与世界报道（USN&WR）的年度初级保健医学院（PCMS）排名吸引了相当多的关注，但它的测量属性尚未公布。作者研究了 PCMS 排名及其衍生的 PCMS 得分的短期稳定性，以及学校排名的短期波动。
作者利用已发表的数据和方法，重建了 2009-2012 年的 PCMS 得分和排名。他们使用混合效应模型评估 PCMS 得分和排名的校内、年内可靠性（短期稳定性），得出组内相关系数（ICC）。他们将短期波动定义为四年研究期间学校排名的中位数校内范围。
重建的 PCMS 得分与四年的发表得分高度相关（皮尔逊相关系数≥98.9%）。大多数学校的平均年度 PCMS 得分在得分分布的中心附近紧密聚集。PCMS 得分和排名的 ICC 分别为 94%和 90%。在研究期间，最好和最差排名之间的中位数差异为 4，排名在 1 到 20 之间的 18 所学校中，而其他 89 所学校的中位数差异为 17（P<.001，Kruskal-Wallis 检验）。
USN&WR PCMS 得分和排名的短期稳定性相当好。然而，PCMS 排名的短期波动较大，特别是在平均排名低于前 20 的学校中。这种可变性大于实际培训质量变化的合理归因。这些发现引发了对排名有效性和有用性的质疑。
