Ames S L, Gallaher P E, Sun P, Pearce S, Zogg J B, Houska B R, Leigh B C, Stacy A W
University of California, Los Angeles, California and University of Southern California, Los Angeles, California, USA.
Behav Res Methods. 2005 Aug;37(3):470-9. doi: 10.3758/bf03192716.
A Web-based coding application was designed to improve coding efficiency and to provide a systematic means of evaluating responses to open-ended assessments. The system was developed for use by multiple raters to assign open-ended responses to predetermined categories. The application provides a software environment for efficiently supervising the work of coders and evaluating the quality of the coding by (1) systematically presenting open-ended responses to coders, (2) tracking each coder's categorized responses, and (3) assessing interrater consistency at any time in order to identify coders in need of further training. In addition, the application can be set to automatically assign repeated responses to categories previously identified as appropriate for those responses. To evaluate the efficacy of the coding application and to determine the statistical reliability of coding open-ended data within this application, we examined data from two empirical studies. The results demonstrated substantial interrater agreement on items assigned to various categories across free and controlled association tasks. Overall, this new coding application provides a feasible method of reliably coding open-ended data and makes the task of coding these data more manageable.
