The rapid advance of large-scale atlas-level single cell RNA sequences and single-cell chromatin accessibility data provide extraordinary avenues to broad and deep insight into complex biological mechanism. Leveraging the datasets and transfering labels from scRNA-seq to scATAC-seq will empower the exploration of single-cell omics data. However, the current label transfer methods have limited performance, largely due to the lower capable of preserving fine-grained cell populations and intrinsic or extrinsic heterogeneity between datasets. Here, we present a robust deep transfer model based graph convolutional network, scTGCN, which achieves versatile performance in preserving biological variation, while achieving integration hundreds of thousands cells in minutes with low memory consumption. We show that scTGCN is powerful to the integration of mouse atlas data and multimodal data generated from APSA-seq and CITE-seq. Thus, scTGCN shows high label transfer accuracy and effectively knowledge transfer across different modalities.