OBJECTIVE: Efficient and effective fidelity measurement is necessary to improve rigor and reduce burden of motivational interviewing (MI) implementation studies for both fidelity outcomes and quality improvement strategies. This article reports on such a measure developed with rigorous methodology and tested in community-based substance abuse treatment settings. METHODS: This scale development study analyzed data from a National Institute on Drug Abuse study testing the Leadership and Organizational Change for Implementation (LOCI) strategy. Utilizing item response theory (IRT) methods and Rasch modeling, we analyzed coded recordings (N = 1089) of intervention sessions of 238 providers from 60 substance use treatment clinics across nine agencies in an implementation trial focusing on MI. RESULTS: These methods yielded a reliable and valid 12-item scale representing single construct dimensionality, strong item-session maps, good rating scale functionality, and item fit. Separation reliability and absolute agreement for adjacent categories was high. No items were significantly misfitting, though one was borderline. The LOCI community providers were less likely to be scored in the advanced competence range and items were more difficult compared to the original development sample. CONCLUSIONS: The 12-item Motivational Interviewing Coach Rating Scale (MI-CRS) showed excellent performance in a large sample of community-based substance use treatment providers using real recordings. The MI-CRS is the first efficient and effective fidelity measure appropriate for diverse ethnic groups, with interventions that are MI only or interventions that integrate MI with other treatments, and with adolescents and adults. Follow-up coaching by trained supervisors may be needed for community-based providers to achieve the highest level of MI competence.