Accurate volumetric segmentation of brain tumors and tissues is beneficial for quantitative brain analysis and brain disease identification in multi-modal Magnetic Resonance (MR) images. Nevertheless, due to the complex relationship between modalities, 3D Fully Convolutional Networks (3D FCNs) using simple multi-modal fusion strategies hardly learn the complex and nonlinear complementary information between modalities. Meanwhile, the indiscriminative feature aggregation between low-level and high-level features easily causes volumetric feature misalignment in 3D FCNs. On the other hand, the 3D convolution operations of 3D FCNs are excellent at modeling local relations but typically inefficient at capturing global relations between distant regions in volumetric images. To tackle these issues, we propose an Aligned Cross-Modality Interaction Network (ACMINet) for segmenting the regions of brain tumors and tissues from MR images. In this network, the cross-modality feature interaction module is first designed to adaptively and efficiently fuse and refine multi-modal features. Secondly, the volumetric feature alignment module is developed for dynamically aligning low-level and high-level features by the learnable volumetric feature deformation field. Thirdly, we propose the volumetric dual interaction graph reasoning module for graph-based global context modeling in spatial and channel dimensions. Our proposed method is applied to brain glioma, vestibular schwannoma, and brain tissue segmentation tasks, and we performed extensive experiments on BraTS2018, BraTS2020, Vestibular Schwannoma, and iSeg-2017 datasets. Experimental results show that ACMINet achieves state-of-the-art segmentation performance on all four benchmark datasets and obtains the highest DSC score of hard-segmented enhanced tumor region on the validation leaderboard of the BraTS2020 challenge.