The processes underlying dynamic changes in human behavior during real situations contain much irrelevant information and represent a key issue facing neuroscientists. Although the roles played by the frontal cortex in this switching behavior have been well documented, little is known regarding how neural pathways governing sensorimotor associations accomplish such a switch. We addressed this question by recording activities of middle temporal (MT) neurons in monkeys switching between direction versus depth discrimination tasks. Although the monkeys successfully switched between the tasks, neural sensitivity did not change as a function of task. More importantly, neurons that signaled the same motor output showed trial-to-trial covariation between neuronal responses and perceptual judgments during both tasks, whereas neurons that signaled the opposite motor output showed no covariation in either task. These results suggest that task switching is accomplished via communication from distinct populations of neurons when sensorimotor associations switch within a short time period.