The purpose of this study is to examine the psychometric properties of an instrument to measure word problem solving skills in mathematics related to speed with 706 sixth grade Chinese and Singaporean students. Rasch measurement models were applied to examine the reliability, unidimensionality, rating scale functioning, item difficulty, and person difficulty. The differential item functioning (DIF) analysis was also performed to examine the differences in item difficulty estimates between Chinese and Singaporean students. Results suggest that the data satisfied the unidimensionality requirements of the Rasch model and that most of the item difficulty measures aligned the person ability distribution. The instrument demonstrated adequate reliability. The fit statistics were within acceptable limits for the vast majority of items, with a few exceptions. The rating scale structure functioned properly although the middle categories had very few observations. Deleting misfitting cases and collapsing middle categories slightly improved the psychometric properties. DIF analysis revealed that four items were more difficult for Chinese students whereas two other items were more difficult for the Singaporean students. Results also indicated that the Chinese participants scored higher than the Singaporean participants for 11 of the 14 items and the Singaporean students scored higher than their Chinese cohorts in the other 3 items. The validation of this instrument has implications for the teaching and learning of mathematical word problems in practice.