Adhikari Santusta, Bastola Anup, Shrestha Reekesh, Kumar Khanal Narendra, Trap Birna
USAID Medicines, Technologies, and Pharmaceutical Services (MTaPS), Kathmandu, Nepal.
Management Sciences for Health, Arlington, Virginia, USA.
J Pharm Policy Pract. 2025 Mar 24;18(1):2477098. doi: 10.1080/20523211.2025.2477098. eCollection 2025.
Nepal's Ministry of Health and Population piloted the Supervision, Performance Assessment, and Recognition Strategy (SPARS), a multipronged strategy to improve medicines management practices in health facilities. Medicines management supervisors (MMS) periodically assess facility performance using 25 indicators grouped into 5 domains - prescribing, dispensing, ordering and reporting quality and stock and storage management - and provide targeted on-the-job coaching. This study aimed to determine inter-rater reliability (IRR) and validity for SPARS indicators in Nepali context.
We assessed IRR and validity scores with three rater teams assessing medicines management in six public health posts. Each team of three MMS and one technical advisor evaluated two facilities, resulting in 24 total assessments. We calculated the mean and median validity and IRR scores for SPARS overall, and by domain and indicator. Acceptable scores were mean validity and IRR scores ≥75%, moderately acceptable were 50-75%, and unacceptable were <50%.
The mean overall IRR (60%) and validity (74%) of all 25 SPARS indicators were significantly different ( < 0.003) but not by domain. Three domains related to dispensing, ordering and reporting, and storage had acceptable mean validity scores, while the prescribing quality domain had an unacceptable mean IRR score. We found significant difference ( < 0.0001) between the indicator IRR and validity acceptability scores. Four (16%) and 12 (48%) of the SPARS indicators had acceptable reproducibility and validity, respectively; 8 (33%) IRR scores and 1 (4%) validity score were unacceptable. Simple indicators (67%) had significantly ( < 0.02) higher reproducibility compared to complex indicators (42%), and higher validity of 80% and 72%, respectively, which was not significant.
This study demonstrated that the SPARS indicators produce moderately accurate findings. Validity was significantly higher than IRR and both need improvement, especially for complex indicators. As such, future SPARS implementation should endeavour to include validity and IRR assessments to help improve the indicators.
尼泊尔卫生与人口部试行监督、绩效评估与认可战略（SPARS），这是一项旨在改善医疗机构药品管理实践的多管齐下的战略。药品管理监督员（MMS）使用25项指标定期评估机构绩效，这些指标分为5个领域——处方、调配、订购与报告质量以及库存与储存管理，并提供有针对性的在职培训。本研究旨在确定尼泊尔背景下SPARS指标的评分者间信度（IRR）和效度。
我们由三个评分者团队对六个公共卫生岗位的药品管理进行评估，以此来评估IRR和效度得分。每个由三名MMS和一名技术顾问组成的团队评估两个机构，共进行了24次评估。我们计算了SPARS总体以及按领域和指标划分的平均和中位数效度及IRR得分。可接受分数为平均效度和IRR得分≥75%，中度可接受为50 - 75%，不可接受为<50%。
所有25项SPARS指标的总体平均IRR（60%）和效度（74%）存在显著差异（<0.003），但按领域划分无显著差异。与调配、订购与报告以及储存相关的三个领域的平均效度得分可接受，而处方质量领域的平均IRR得分不可接受。我们发现指标IRR和效度可接受性得分之间存在显著差异（<0.0001）。分别有4项（16%）和12项（48%）SPARS指标具有可接受的再现性和效度；8项（33%）IRR得分和1项（4%）效度得分不可接受。简单指标（67%）的再现性显著高于复杂指标（42%）（<0.02），效度分别为80%和72%，但差异不显著。
本研究表明，SPARS指标得出的结果准确性适中。效度显著高于IRR，两者都需要改进，尤其是对于复杂指标。因此，未来SPARS的实施应努力纳入效度和IRR评估，以帮助改进指标。
