OBJECTIVES: Pain assessment is essential to tailor intensive care of neonates. The present focus is on acute procedural pain; assessment of pain of longer duration remains a challenge. We therefore tested a modified version of the COMFORT-behavior scale-named COMFORTneo-for its psychometric qualities in the Neonatal Intensive Care Unit setting. METHODS: In a clinical observational study, nurses assessed patients with COMFORTneo and Numeric Rating Scales (NRS) for pain and distress, respectively. Interrater reliability, concurrent validity, and sensitivity to change were calculated as well as sensitivity and specificity for different cut-off scores for subsets of patients. RESULTS: Interrater reliability was good: median linearly weighted Cohen kappa 0.79. Almost 3600 triple ratings were obtained for 286 neonates. Internal consistency was good (Cronbach alpha 0.84 and 0.88). Concurrent validity was demonstrated by adequate and good correlations, respectively, with NRS-pain and NRS-distress: r=0.52 (95% confidence interval 0.44-0.59) and r=0.70 (95% confidence interval 0.64-0.75). COMFORTneo cut-off scores of 14 or higher (score range is 6 to 30) had good sensitivity and specificity (0.81 and 0.90, respectively) using NRS-pain or NRS-distress scores of 4 or higher as criterion. DISCUSSION: The COMFORTneo showed preliminary reliability. No major differences were found in cut-off values for low birth weight, small for gestational age, neurologic impairment risk levels, or sex. Multicenter studies should focus on establishing concurrent validity with other instruments in a patient group with a high probability of ongoing pain.