Short-term memory for sequences of verbal items such as written words is reliably impaired by task-irrelevant background sounds, a phenomenon known as the "Irrelevant Sound Effect" (ISE). Different theoretical accounts have been proposed to explain the mechanisms underlying the ISE. Some of these assume specific interference between obligatory sound processing and phonological or serial order representations generated during task performance, whereas other posit that background sounds involuntarily divert attention away from the focal task. To explore the roles of phonological processing, serial order retention, and attention control, we analyzed the effects of environmental non-speech sounds and unfamiliar speech on serial recall of verbal items (pictures representing German nouns) and spatial items (dot locations) in children (n = 137) and adults (n = 98). In the verbal task, both age groups were equally affected by background sounds, with speech impairing recall more than environmental sounds. In the spatial task, no ISE was found in adults and fourth graders, but third graders exhibited significant performance impairment from both sounds. There was no habituation to the sound effects across the experimental trials. The findings indicate that both specific interference and attention capture may contribute to the ISE, with the impact of attention capture potentially decreasing with age.