Left ventricular diastolic dyfunction detection is particularly important in cardiac function screening. This paper proposed a phonocardiogram (PCG) transfer learning-based CatBoost model to detect diastolic dysfunction noninvasively. The Short-Time Fourier Transform (STFT), Mel Frequency Cepstral Coefficients (MFCCs), S-transform and gammatonegram were utilized to perform four different representations of spectrograms for learning the representative patterns of PCG signals in two-dimensional image modality. Then, four pre-trained convolutional neural networks (CNNs) such as VGG16, Xception, ResNet50 and InceptionResNetv2 were employed to extract multiple domain-specific deep features from PCG spectrograms using transfer learning, respectively. Further, principal component analysis and linear discriminant analysis (LDA) were applied to different feature subsets, respectively, and then these different selected features are fused and fed into CatBoost for classification and performance comparison. Finally, three typical machine learning classifiers such as multilayer perceptron, support vector machine and random forest were employed to compared with CatBoost. The hyperparameter optimization of the investigated models was determined through grid search. The visualized result of the global feature importance showed that deep features extracted from gammatonegram by ResNet50 contributed most to classification. Overall, the proposed multiple domain-specific feature fusion based CatBoost model with LDA achieved the best performance with an area under the curve of 0.911, accuracy of 0.882, sensitivity of 0.821, specificity of 0.927, F1-score of 0.892 on the testing set. The PCG transfer learning-based model developed in this study could aid in diastolic dysfunction detection and could contribute to non-invasive evaluation of diastolic function.