Selective attention modulates neural activity in the visual system both in the presence and in the absence of visual stimuli. When subjects direct attention to a particular location in a visual scene in anticipation of the stimulus onset, there is an increase in baseline activity. How do such baseline increases relate to the attentional modulation of stimulus-driven activity? Using functional magnetic resonance imaging, we demonstrate that baseline increases related to the expectation of motion or color stimuli at a peripheral target location do not predict the modulation of neural responses evoked by these stimuli when attended. In areas such as MT and TEO that were more effectively activated by one stimulus type than the other, attentional modulation of visually evoked activity depended on the stimulus preference of a visual area and was stronger for the effective than for the noneffective stimulus. In contrast, baseline increases did not reflect the stimulus preference of a visual area. Rather, these signals were shown to be spatially specific and appeared to be dominated by the location information and not by the feature information of the cue with the experimental paradigms under study. These findings provide evidence that baseline increases in visual cortex during cue periods do not reflect the activation of a memory template that includes particular stimulus properties of the expected target, but rather carry information about the location of an expected target stimulus. In addition, when the stimulus contained both color and motion, an object-based attention effect was observed, with significant attentional modulation in the area that responded preferentially to the unattended feature.