Enhancing the accuracy of long-term time series forecasting is a crucial task across various fields. Recently, many studies have employed the Discrete Fourier Transform (DFT) to convert time series into the frequency domain for forecasting, as it provides a compact and efficient representation, enabling the capture of deep cyclical patterns and global trends that are often difficult to identify in the time domain. However, the frequency domain information in time series has yet to be fully explored due to three main reasons: they rely on a fixed-resolution DFT, which restricts them to a single frequency resolution and leads to the loss of essential information; they predominantly focus on global dependencies while overlooking local temporal details; they remain highly sensitive to noise in the time series, limiting the model's ability to capture stable patterns. In this paper, we propose a novel lightweight architecture (FreMixer) that operates entirely in the frequency domain. Firstly, we introduce a multi resolution segmentation mechanism in the frequency domain, enabling features represented at different resolutions to complement each other, effectively overcoming the sparse resolution limitations of DFT in the frequency spectrum. Secondly, we comprehensively extract the frequency information by employing a dual branch architecture that simultaneously captures both global and local features at each frequency resolution, providing a more comprehensive representation of temporal patterns. Moreover, we propose a noise insensitive loss function ArcTanLoss that reduces overfitting to outliers. Extensive experiments conducted on seven different datasets have validated the effectiveness of our proposed model and loss function.