Due to the overlapping depressive symptomatology with major depressive disorder (MDD), 60% of patients with bipolar disorder (BD) are initially misdiagnosed, calling for the definition of reliable biomarkers that can support the diagnostic process. Here, we optimized a machine learning pipeline for the differentiation between depressed BD and MDD patients based on multimodal structural neuroimaging features. Diffusion tensor imaging (DTI) and T1-weighted magnetic resonance imaging (MRI) data were acquired for 282 depressed BD (n = 180) and MDD (n = 102) patients. Images were preprocessed to obtain axial (AD), radial (RD), mean (MD) diffusivity, fractional anisotropy (FA), and voxel-based morphometry (VBM) maps. Each feature was entered separately into a 5-fold nested cross-validated predictive pipeline differentiating between BD and MDD patients, comprising: confound regression for nuisance variables removal, feature standardization, principal component analysis for feature reduction, and an elastic-net penalized regression. The DTI-based models reached accuracies ranging from 75% to 78%, whereas the VBM model reached 61% of accuracy. All the models were significantly different from a null model distribution at a 5000-permutation test. A 5000 bootstrap procedure revealed that widespread differences drove the classification, with BD patients associated to overall higher values of AD and FA, and grey matter volumes. Our results suggest that structural neuroimaging, in particular white matter microstructure and grey matter volumes, may be able to differentiate between MDD and BD patients with good predictive accuracy, being significantly higher than chance-level.