Gadolinium-based contrast agents are commonly used in cardiac magnetic resonance (CMR) imaging to characterize myocardial scar tissue. Recent works using deep learning have shown the promise of contrast-free short-axis cine images to detect scars based on wall motion abnormalities (WMA) in ischemic patients. However, WMA can occur in patients without a scar. Moreover, the presence of a scar may not always be accompanied by WMA, particularly in non-ischemic heart disease, posing a significant challenge in detecting scars in such cases. To overcome this limitation, we propose a novel deep spatiotemporal residual attention network (ST-RAN) that leverages temporal and spatial information at different scales to detect scars in both ischemic and non-ischemic heart diseases. Our model comprises three primary components. First, we develop a novel factorized 4D (3D+time) convolutional layer that extracts 3D spatial features of the heart and a deep 1D kernel in the temporal direction to extract heart motion. Secondly, we enhance the power of the 4D (3D+time) layer with spatiotemporal attention to extract rich whole-heart features while tracking the long-range temporal relationship between the frames. Lastly, we introduce a residual attention block that extracts spatial and temporal features at different scales to obtain global and local motion features and to detect subtle changes in contrast related to scar. We train and validate our model on a large dataset of 3000 patients who underwent clinical CMR with various indications and different field strengths (1.5T, 3T) from multiple vendors (GE, Siemens) to demonstrate the generalizability and robustness of our model. We show that our model works on both ischemic and non-ischemic heart diseases outperforming state-of-the-art methods. Our code is available at https://github.com/HMS-CardiacMR/Myocardial_Scar_Detection.