White matter impairments caused by gliomas can lead to functional disorders. In this study, we predicted aphasia in patients with gliomas infiltrating the language network using machine learning methods. We included 78 patients with left-hemispheric perisylvian gliomas. Aphasia was graded preoperatively using the Aachen aphasia test (AAT). Subsequently, we created bundle segmentations based on automatically generated tract orientation mappings using TractSeg. To prepare the input for the support vector machine (SVM), we first preselected aphasia-related fiber bundles based on the associations between relative tract volumes and AAT subtests. In addition, diffusion magnetic resonance imaging (dMRI)-based metrics [axial diffusivity (AD), apparent diffusion coefficient (ADC), fractional anisotropy (FA), and radial diffusivity (RD)] were extracted within the fiber bundles' masks with their mean, standard deviation, kurtosis, and skewness values. Our model consisted of random forest-based feature selection followed by an SVM. The best model performance achieved 81% accuracy (specificity = 85%, sensitivity = 73%, and AUC = 85%) using dMRI-based features, demographics, tumor WHO grade, tumor location, and relative tract volumes. The most effective features resulted from the arcuate fasciculus (AF), middle longitudinal fasciculus (MLF), and inferior fronto-occipital fasciculus (IFOF). The most effective dMRI-based metrics were FA, ADC, and AD. We achieved a prediction of aphasia using dMRI-based features and demonstrated that AF, IFOF, and MLF were the most important fiber bundles for predicting aphasia in this cohort.