The lateralization of visuospatial attention has been well investigated and demonstrated to be primarily resulting from unbalanced interaction between interhemispheric fronto-parietal networks in previous studies. Many recent studies of top-down attention have reported the neural signatures of its effects within visual cortex and identified its causal basis. However, the relationship between top-down networks and asymmetric visuospatial attention has not been well studied. In the current study, we aimed to explore the relationship between top-down connectivity and asymmetric visuospatial ability by using repetitive transcranial magnetic stimulation (rTMS) and resting-state functional connectivity (RSFC) analyses. We used rTMS and RSFC to model the virtual lesion to assess the behavioral performances in visuospatial attention shifting and to identify the behavior-related top-down functional connectivities, respectively. Furthermore, we also investigated the top-down connectivity in neglect patients to validate the RSFC findings. RSFC analyses in healthy subjects and neglect patients consistently revealed that asymmetric visuospatial ability and visuospatial neglect were closely related to the bias of top-down functional connectivity between posterior superior parietal lobule (SPL) and V1. Our findings indicate that stronger top-down connectivity has stronger dominance on its corresponding visual field. We argue that an asymmetric top-down network may represent a possible neurophysiological substrate for the ongoing functional asymmetry of visuospatial attention, and its interhemispheric unbalanced interaction could contribute to the clinical manifestations of visuospatial neglect.