Transfer learning is increasingly used to decode multi-class motor imagery tasks. Previous transfer learning ignored the optimizability of the source model, weakened the adaptability to the target domain and limited the performance. This paper first proposes the multi-loss fusion convolutional neural network (MF-CNN) to make an optimizable source model. Then we propose a novel source optimized transfer learning (SOTL), which optimizes the source model to make it more in line with the target domain's features to improve the target model's performance. We transfer the model trained from 16 healthy subjects to 16 stroke patients. The average classification accuracy achieves 51.2 ± 0.17% in the four types of unilateral upper limb motor imagery tasks, which is significantly higher than the classification accuracy of deep learning ( < 0.001) and transfer learning ( < 0.05). In this paper, an MI model from the data of healthy subjects can be used for the classification of stroke patients and can demonstrate good classification results, which provides experiential support for the study of transfer learning and the modeling of stroke rehabilitation training.