Understanding speech in noisy, multi-talker environments is crucial for social communication but becomes increasingly challenging and frustrating as we age. Here, we simulated the acoustic challenges of multi-talker listening and found that adults over 50 years old (N = 76) recognized speech more slowly, less accurately, and less consistently than younger adults (N = 107). While peripheral hearing status accounted for average differences in speech intelligibility by age, it did not account for moment-to-moment variability in speed and accuracy - fluctuations central to the frustration experienced by older listeners in challenging environments. We hypothesized that age-related changes in brain arousal systems might account for the fluctuant "noise" in speech processing observed in older listeners. To isolate the contribution of arousal state independent of hearing status and cognitive load, we measured the pre-stimulus pupil-indexed arousal state (PPAS) immediately prior to speech onset. Older - but not younger - adults exhibited a striking inverted-U relationship between PPAS and speech recognition accuracy. Notably, pupil-indexed listening effort measured seconds later during speech encoding was not associated with trial-to-trial performance. Moreover, older adults exhibited altered arousal regulation, occupying a lower PPAS extremum not observed in younger listeners that was specifically associated with performance deficits and subjective listening difficulties reported in hearing health questionnaires. These findings show that age-related changes in central arousal states interact with peripheral hearing status to offer a more complete explanation for why older adults find speech processing in social setting so challenging.