Spatial attention improves performance at attended locations and correspondingly modulates firing rates of cortical neurons. The size of these behavioral and neuronal effects depends on the difficulty of the task performed at the attended location. Psychological theorists have attributed this to a tighter focus of a fixed amount of processing resource at the attended location, but the effects of task difficulty on the distribution of neuronal effects of attention across the visual field have not been fully explored. We trained rhesus monkeys to do a detection task in which difficulty and spatial attention were manipulated independently. Probe stimuli were used to measure behavioral performance in different conditions of attention and difficulty. Animals performed better at attended locations and this advantage increased with difficulty, consistent with data from human psychophysics. Neuronal modulation by spatial attention was larger with greater difficulty. In two animals, increasing difficulty caused a modest increase in neuronal responses to visual stimuli regardless of the locus of spatial attention. In a third animal, which was previously trained to ignore multiple distracting stimuli, increasing task difficulty increased responses at the focus of attention and suppressed responses away from the focus of attention. The results show that difficulty can modulate effects of spatial attention in V4; it can alter the distribution of sensory responses across the visual scene in ways that may depend on the subject's behavioral strategy.