Despite the traditional view of Schizophrenia (SZ) and Bipolar disorder (BD) as separate diagnostic categories, the validity of such a categorical approach is challenging. In recent years, the hypothesis of a continuum between Schizophrenia (SZ) and Bipolar disorder (BD), postulating a common pathophysiologic mechanism, has been proposed. Although appealing, this unifying hypothesis may be too simplistic when looking at cognitive and affective differences these patients display. In this paper, we aim to test an expanded version of the continuum hypothesis according to which the continuum extends over three clusters: the psychotic, the cognitive, and the affective. We applied an innovative approach known as Source-based Morphometry (SBM) to the structural images of 46 individuals diagnosed with SZ, 46 with BD and 66 healthy controls (HC). We also analyzed the psychological profiles of the three groups using cognitive, affective, and clinical tests. At a neural level, we found evidence for a shared psychotic core in a distributed network involving portions of the medial parietal and temporo-occipital areas, as well as parts of the cerebellum and the middle frontal gyrus. We also found evidence of a cognitive core more compromised in SZ, including alterations in a fronto-parietal circuit, and mild evidence of an affective core more compromised in BD, including portions of the temporal and occipital lobes, cerebellum, and frontal gyrus. Such differences were confirmed by the psychological profiles, with SZ patients more impaired in cognitive tests, while BD in affective ones. On the bases of these results we put forward an expanded view of the continuum hypothesis, according to which a common psychotic core exists between SZ and BD patients complemented by two separate cognitive and affective cores that are both impaired in the two patients' groups, although to different degrees.