This study explored the taxonomy of cognitive impairment within temporal lobe epilepsy and characterized the sociodemographic, clinical and neurobiological correlates of identified cognitive phenotypes. 111 temporal lobe epilepsy patients and 83 controls (mean ages 33 and 39, 57% and 61% female, respectively) from the Epilepsy Connectome Project underwent neuropsychological assessment, clinical interview, and high resolution 3T structural and resting-state functional MRI. A comprehensive neuropsychological test battery was reduced to core cognitive domains (language, memory, executive, visuospatial, motor speed) which were then subjected to cluster analysis. The resulting cognitive subgroups were compared in regard to sociodemographic and clinical epilepsy characteristics as well as variations in brain structure and functional connectivity. Three cognitive subgroups were identified (intact, language/memory/executive function impairment, generalized impairment) which differed significantly, in a systematic fashion, across multiple features. The generalized impairment group was characterized by an earlier age at medication initiation (P < 0.05), fewer patient (P < 0.001) and parental years of education (P < 0.05), greater racial diversity (P < 0.05), and greater number of lifetime generalized seizures (P < 0.001). The three groups also differed in an orderly manner across total intracranial (P < 0.001) and bilateral cerebellar cortex volumes (P < 0.01), and rate of bilateral hippocampal atrophy (P < 0.014), but minimally in regional measures of cortical volume or thickness. In contrast, large-scale patterns of cortical-subcortical covariance networks revealed significant differences across groups in global and local measures of community structure and distribution of hubs. Resting-state fMRI revealed stepwise anomalies as a function of cluster membership, with the most abnormal patterns of connectivity evident in the generalized impairment group and no significant differences from controls in the cognitively intact group. Overall, the distinct underlying cognitive phenotypes of temporal lobe epilepsy harbor systematic relationships with clinical, sociodemographic and neuroimaging correlates. Cognitive phenotype variations in patient and familial education and ethnicity, with linked variations in total intracranial volume, raise the question of an early and persisting socioeconomic-status related neurodevelopmental impact, with additional contributions of clinical epilepsy factors (e.g., lifetime generalized seizures). The neuroimaging features of cognitive phenotype membership are most notable for disrupted large scale cortical-subcortical networks and patterns of functional connectivity with bilateral hippocampal and cerebellar atrophy. The cognitive taxonomy of temporal lobe epilepsy appears influenced by features that reflect the combined influence of socioeconomic, neurodevelopmental and neurobiological risk factors.