The brightness and color of a surface depends on its contrast with nearby surfaces. For example, a gray surface can appear very light when surrounded by a black surface or dark when surrounded by a white surface. Some theories suggest that perceived surface brightness and color is represented explicitly by neural signals in cortical visual field maps; these neural signals are not initiated by the stimulus itself but rather by the contrast signals at the borders. Here, we use functional magnetic resonance imaging (fMRI) to search for such neural "filling-in" signals. Although we find the usual strong relationship between local contrast and fMRI response, when perceived brightness or color changes are induced by modulating a surrounding field, rather than the surface itself, we find there is no corresponding local modulation in primary visual cortex or other nearby retinotopic maps. Moreover, when we model the obtained fMRI responses, we find strong evidence for contributions of both local and long-range edge responses. We argue that such extended edge responses may be caused by neurons previously identified in neurophysiological studies as being brightness responsive, a characterization that may therefore need to be revised. We conclude that the visual field maps of human V1 and V2 do not contain filled-in, topographical representations of surface brightness and color.