Investigating the factors influencing rice grain yield (GY) is critical for optimizing nitrogen (N) management and enhancing resource use efficiency in rice cultivation. However, few studies have comprehensively investigated the factors affecting rice GY, considering an entire influence chain encompassing rice N uptake, growth indicators, and GY components. In this study, field experiment with six different N fertilizer rates (0, 60, 120, 180, 225, and 300 kg N ha, i.e., N0, N60, N120, N180, N225, and N300) was conducted in the Jianghan Plain in the Middle Reaches of the Yangtze River, China, to comprehensively elucidate the factors influencing rice GY from aspects of rice N uptake, growth indicators, and GY components and determine the optimal N fertilizer rate. The results showed that rice GY and N uptake initially increased and then either stabilized or declined with higher N fertilizer rate, while apparent N loss escalated with increased N fertilizer rate. The application of N fertilizer significantly promoted the increase in straw N uptake, which was significantly positively correlated with growth indicators ( < 0.05). Among all GY components, panicle number per hill was the most significant positive factor influencing rice GY, and it was significantly positively correlated with all rice growth indicators ( < 0.05). In addition, N180 was the optimal N fertilizer rate, ensuring more than 95% of maximum GY and reducing N loss by 74% and 39% compared to N300, respectively. Meanwhile, the average N balance for N180 remained below 60 kg N ha. In conclusion, optimizing the N fertilizer application in paddy fields can effectively maintain stable rice GY and minimize environmental pollution.