Although elderly people with amnestic mild cognitive impairment (aMCI) have been found to show impaired behavioral performance in task switching, no research has yet explored the electrophysiological mechanisms and the potential correlation between physical fitness and neurocognitive (i.e., behavioral and electrophysiological) performance in aMCI. The present study was thus aimed to examine whether there are differences in electrophysiological (i.e., event-related potential) performance between aMCI participants and controls when performing a task-switching paradigm, and to investigate the role of physical fitness in the relationship between neurocognitive performance and aMCI. Sixty participants were classified into aMCI (n = 30) and control (n = 30) groups, and performed a task-switching paradigm with concomitant electrophysiological recording, as well as underwent senior functional physical fitness tests. The aMCI group showed comparable scores on most parts of the physical fitness tests, but reduced lower body flexibility and VO2max as compared to the control group. When performing the task-switching paradigm, the aMCI group showed slower reaction times in the heterogeneous condition and larger global switching costs, although no significant difference was observed in accuracy rates between the two groups. In addition, the aMCI group showed significantly prolonged P3 latencies in the homogeneous and heterogeneous conditions, and a smaller P3 amplitude only in the heterogeneous condition. The level of cardiorespiratory fitness was significantly correlated with P3 amplitude in the aMCI group, particularly in the heterogeneous condition of the task-switching paradigm. These results show that the aMCI group exhibited abnormalities in their neurocognitive performance when performing the task-switching paradigm and such a deficit was likely associated with reduced cardiorespiratory fitness, which was shown to be the important predictor of neurocognitive performance.