The left ventricular ejection fraction is of significant importance for the early identification and diagnosis of cardiac disease. However, estimation of the left ventricular ejection fraction with consistently reliable and high accuracy remains a great challenge, owing to the high variability of cardiac structures and the complexity of the temporal dynamics in the cardiac magnetic resonance imaging sequences. The popular methods of left ventricular ejection fraction estimation rely on the left ventricular volume. Thus, strong prior knowledge is often necessary, impeding the ease of use of the existing methods as clinical tools. In this study, we propose a cardiac cycle feature learning architecture for achieving an accurate and reliable estimation of the left ventricular ejection fraction. The proposed method constructs a cardiac cycle extraction module that generates and analyzes an optical flow to obtain the cardiac cycle of all images, a motion feature fusion and extraction module for temporal modeling of the cardiac sequences, and a fully connected regression module for achieving a direct estimation. Experiments on 2900 left ventricle segments of 145 subjects from short-axis magnetic resonance imaging sequences of multiple lengths prove that our proposed method achieves reliable performance (correlation coefficient: 0.946; mean absolute error 2.67; standard deviation: 3.23). As compared with the current state-of-the-art method, our proposed method improves the performance by approximately 3% insofar as the mean absolute error. As the first solution for estimating the left ventricular ejection fraction directly, our proposed method demonstrates great potential for future clinical applications.