(1) Background: The literature on the effects of high-grade glioma (HGG) growth on cognition is still scarce. (2) Method: A consecutive series of 85 patients with HGG involving the left hemisphere underwent an extended neuropsychological evaluation prior to surgery. Voxel-based lesion-symptom mapping (VLSM) was used to identify regions related to cognitive performance. (3) Results: The patients' mean level of pre-surgery accuracy was overall high. They showed the greatest difficulties in language with tasks such as naming (42.1% of patients impaired on nouns and 61.4% on verbs), reading (36.3% on words and 32.7% on pseudo-words), auditory lexical decisions (43.9%) and writing (41.3%) being most frequently impaired. VLSM analysis revealed anatomically separated areas along the temporal cortex and the white matter related to impairments on the different tasks, with voxels commonly shared by all tasks restricted to a small region in the ventral superior and middle temporal gyrus. (4) Conclusions: High-grade glioma affects cognition; nonetheless, lesions do not cause diffuse deficits but selectively impact the different language sub-domains along the ventral stream and the dorsal stream for language processing.