We contrasted the neuroanatomical substrates of sub-acute and chronic visuospatial deficits associated with different aspects of unilateral neglect using computed tomography scans acquired as part of routine clinical diagnosis. Voxel-wise statistical analyses were conducted on a group of 160 stroke patients scanned at a sub-acute stage. Lesion-deficit relationships were assessed across the whole brain, separately for grey and white matter. We assessed lesions that were associated with behavioural performance (i) at a sub-acute stage (within 3 months of the stroke) and (ii) at a chronic stage (after 9 months post stroke). Allocentric and egocentric neglect symptoms at the sub-acute stage were associated with lesions to dissociated regions within the frontal lobe, amongst other regions. However the frontal lesions were not associated with neglect at the chronic stage. On the other hand, lesions in the angular gyrus were associated with persistent allocentric neglect. In contrast, lesions within the superior temporal gyrus extending into the supramarginal gyrus, as well as lesions within the basal ganglia and insula, were associated with persistent egocentric neglect. Damage within the temporo-parietal junction was associated with both types of neglect at the sub-acute stage and 9 months later. Furthermore, white matter disconnections resulting from damage along the superior longitudinal fasciculus were associated with both types of neglect and critically related to both sub-acute and chronic deficits. Finally, there was a significant difference in the lesion volume between patients who recovered from neglect and patients with chronic deficits. The findings presented provide evidence that (i) the lesion location and lesion size can be used to successfully predict the outcome of neglect based on clinical CT scans, (ii) lesion location alone can serve as a critical predictor for persistent neglect symptoms, (iii) wide spread lesions are associated with neglect symptoms at the sub-acute stage but only some of these are critical for predicting whether neglect will become a chronic disorder and (iv) the severity of behavioural symptoms can be a useful predictor of recovery in the absence of neuroimaging findings on clinical scans. We discuss the implications for understanding the symptoms of the neglect syndrome, the recovery of function and the use of clinical scans to predict outcome.