Mental fatigue is defined as a decline in the ability and efficiency of mental activities. A lot of research suggests that the transition from alertness to fatigue is accompanied by alterations in correlation patterns among various brain regions. However, conventional methods for detecting mental fatigue seldom emphases inter-channel connectivity in the spatial domain. To fill this gap, this paper explores the spatial inter-channel connectivity in alertness and fatigue, employing spectral graph convolutional networks (GCN) for mental fatigue detection. We utilized Pearson correlation coefficients (PCC) to establish temporal connections and magnitude-squared coherence (MSC) for spectral connections. Topological features of the brain network were then analysed. To enhance the learning of spatial inter-channel connectivity, a dual-graph strategy transforms edge features into node features, serving as inputs to the spectral GCN. By simultaneously learning PCC and MSC features, the model results indicate significant differences in some brain network characteristics between alert and fatigue states. It confirms that the synchronicity of brain operations differs in the alert state compared to mental fatigue, and indicates that fatigue states can influence correlation patterns among different brain regions. Our approach is evaluated on a self-designed experimental dataset containing 7 subjects, demonstrating a classification accuracy of 89.59 % in group-level experiments and 95.24 % at the subject level. Additionally, on the public dataset SEED-VIG containing 23 subjects, our method achieves an accuracy of 86.58 %. In summary, this paper proposes a neural network approach based on a dynamic functional connectivity network. The network integrates both temporal and spectral connections with the goal of simultaneously learning spatial inter-channel connectivity in time and frequency domains. This effectively accomplishes fatigue state detection, highlighting that fatigue significantly influences correlations among different brain regions.