We used functional magnetic resonance imaging (fMRI) to investigate whether the neural systems for nonspatial visual and auditory working memory exhibits a functional dissociation. The subjects performed a delayed recognition task for previously unfamiliar faces and voices and an audiovisual sensorimotor control task. During the initial sample and subsequent test stimulus presentations, activation was greater for the face than for the voice identity task bilaterally in the occipitotemporal cortex and, conversely, greater for voices than for faces bilaterally in the superior temporal sulcus/gyrus (STS/STG). Ventral prefrontal regions were activated by both memory delays in comparison with the control delays, and there was no significant difference in direct voxelwise comparisons between the tasks. However, further analyses showed that there was a subtle difference in the functional topography for two delay types within the ventral prefrontal cortex. Face delays preferentially activate the dorsal part of the ventral prefrontal cortex (BA 44/45) while voice delays preferentially activate the inferior part (BA 45/47), indicating a ventral/dorsal auditory/visual topography within the ventral prefrontal cortex. The results confirm that there is a modality-specific attentional modulation of activity in visual and auditory sensory areas during stimulus presentation. Moreover, within the nonspatial information-type domain, there is a subtle across-modality dissociation within the ventral prefrontal cortex during working memory maintenance of faces and voices.