This parametric functional magnetic resonance imaging (fMRI) study investigates the balance of negative and positive fMRI signals in the brain. A set of visual attention (VA) and working memory (WM) tasks with graded levels of difficulty was used to deactivate separate but overlapping networks that include the frontal, temporal, occipital, and limbic lobes; regions commonly associated with auditory and emotional processing. Brain activation (% signal change and volume) was larger for VA tasks than for WM tasks, but deactivation was larger for WM tasks. Load-related increases of blood oxygenation level-dependent (BOLD) responses for different levels of task difficulty cross-correlated strongly in the deactivated network during VA but less so during WM. The variability of the deactivated network across different cognitive tasks supports the hypothesis that global cerebral blood flow vary across different tasks, but not between different levels of task difficulty of the same task. The task-dependent balance of activation and deactivation might allow maximization of resources for the activated network.