Working memory for voice identity and words was studied to investigate whether the neural system underlying extralinguistic and linguistic information processing is dissociated and whether the possible differences in the distribution of activity are related to specific periods of working memory tasks. Separate analyses of task-related activations evoked during the encoding, maintenance, and recognition periods of the memory tasks were performed. During the voice task, the superior temporal, ventral prefrontal and medial frontal cortices were activated in comparison with the control task whereas the word task produced activation in the occipital, parietal, and dorsal prefrontal areas. Direct contrasts between different periods of the tasks indicated that the ventral prefrontal cortex and the right superior temporal sulcus/gyrus were more activated during recognition than encoding and maintenance periods in the voice compared with the word task. In contrast, the right supramarginal gyrus was more active during the recognition than encoding period in the word compared with the voice task. The results suggest that dissociable neural substrates are recruited for processing of linguistic and extralinguistic information during the recognition period of a working memory task.