Immune checkpoint therapy (ICT) has greatly improved the survival of cancer patients in the past few years, but only a small number of patients respond to ICT. To predict ICT response, we developed a multi-modal feature fusion model based on deep learning (MFMDL). This model utilizes graph neural networks to map gene-gene relationships in gene networks to low dimensional vector spaces, and then fuses biological pathway features and immune cell infiltration features to make robust predictions of ICT. We used five datasets to validate the predictive performance of the MFMDL. These five datasets span multiple types of cancer, including melanoma, lung cancer, and gastric cancer. We found that the prediction performance of multi-modal feature fusion model based on deep learning is superior to other traditional ICT biomarkers, such as ICT targets or tumor microenvironment-associated markers. In addition, we also conducted ablation experiments to demonstrate the necessity of fusing different modal features, which can improve the prediction accuracy of the model.