Hearing impairment goes with speech perception difficulties, presumably not only because of poor hearing sensitivity but also because of altered central auditory processing. Critical herein is temporal processing of the speech envelope, mediated by synchronization of neural activity to the envelope modulations. It has been suggested that hearing impairment is associated with enhanced sensitivity to envelope modulations which, in turn, relates to poorer speech perception. To verify this hypothesis, we performed a comparative electrophysiological study in hearing-impaired (HI) and normal-hearing (NH) human listeners of three age groups, investigating neural envelope encoding. HI young and middle-aged adults showed enhanced neural synchronization to envelope modulations relative to NH controls, particularly when stimulus audibility was corrected for. At an older age, the degree of neural synchronization was similar for HI and NH persons, yet HI persons showed a synchronization asymmetry toward the right hemisphere. This study demonstrates that hearing impairment is characterized by changes in the neural encoding of envelope modulations, the nature of which varies with age.