A task set is a configuration of cognitive processes that is actively maintained for subsequent task performance. Single-unit and brain-imaging studies have identified the neural correlates for task sets in the prefrontal cortex. Here I examine whether the neural data obtained thus far are sufficient to explain the behaviors that have been illustrated within the conceptual framework of task sets. I first discuss the selectivity of neural activity in representing a specific task. I then discuss the competitions between neural representations of task sets during task switch. Finally I discuss how, in neural terms, a task set is implemented to facilitate task performance. The processes of representing, updating, and implementing task sets occur in parallel at multiple levels of brain organization. Neural accounts of task sets demonstrate that the brain determines our thoughts and behaviors.