This study aimed at determining how the human brain automatically processes phoneme categories irrespective of the large acoustic inter-speaker variability. Subjects were presented with 450 different speech stimuli, equally distributed across the [a], [i], and [u] vowel categories, and each uttered by a different male speaker. A 306-channel magnetoencephalogram (MEG) was used to record N1m, the magnetic counterpart of the N1 component of the auditory event-related potential (ERP). The N1m amplitude and source locations differed between vowel categories. We also found that the spectrum dissimilarities were reproduced in the cortical representations of the large set of the phonemes used in this study: vowels with similar spectral envelopes had closer cortical representations than those whose spectral differences were the largest. Our data further extend the notion of differential cortical representations in response to vowel categories, previously demonstrated by using only one or a few tokens representing each category.